Migrate data using MMS-MaxCompute(MaxCompute)-阿里云帮助中心

After you complete the target schema and SQL conversion, use MaxCompute Migration Service (MMS) to migrate your data.

Objectives

Configure a GCP service account for MMS with the minimum required BigQuery permissions.
Register a BigQuery data source in MMS and create full and incremental migration jobs.
Map BigQuery partitions to MaxCompute partitions during job configuration.
Monitor job progress, validate row counts, and recover from partial failures.

For most BigQuery-to-MaxCompute migrations, MMS is the recommended approach. It avoids a manual export/import pipeline (BigQuery → GCS → OSS → MaxCompute), which can cause data integrity issues and increase costs at scale.

MMS capabilities

MaxCompute Migration Service (MMS 4.0) is a managed migration tool that connects directly to BigQuery through the BigQuery Storage Read API and writes data to MaxCompute through the Tunnel service.

Key capabilities

Capability	Description
Direct migration	Does not require intermediate storage in GCS or OSS.
Automatic schema conversion	Maps data types automatically according to the rules in Document 3; no manual DDL is required.
Partition-aware migration	Preserves BigQuery time-column and ingestion-time partitions and maps them to MaxCompute `AUTO PARTITIONED BY`.
Full and incremental modes	Migrates an entire project, specific datasets, specific tables, or data filtered by partition.
Configurable concurrency	Supports job-level parallelism and a per-table thread pool.
Bandwidth control	Limits egress throughput to comply with BigQuery quotas or network service-level agreements (SLAs).
Built-in validation	Compares row counts after each table is migrated.
Retry mechanism	Supports job-level and table-level retries to recover from partial failures.
Progress dashboard	Displays table-level status (Pending, Running, Completed, or Failed) and metrics for each table.

When not to use MMS
- Do not use MMS for real-time streaming data, such as BigQuery tables populated by Pub/Sub that require zero-gap capture. Instead, set up a parallel Flink or DataHub ingestion pipeline first. Use MMS to backfill historical data after the cutover.
- If schemas require substantial manual adjustments, such as type casting for downstream consumers, pre-create the target table in MaxCompute and select the Use Existing schema strategy in MMS instead of using auto-generation.

Procedure

Configure a GCP service account

Required IAM roles

Role	Purpose
`roles/bigquery.dataViewer`	Allows MMS to read dataset metadata and table data. Grant this role at the project or dataset level.
`roles/bigquery.jobUser`	Allows MMS to run BigQuery Storage Read API sessions, which are metered as jobs. Grant this role at the project level.

Create using the command line

export GCP_PROJECT="your-project"
export SA_NAME="mms-migration"
export SA_EMAIL="${SA_NAME}@${GCP_PROJECT}.iam.gserviceaccount.com"

gcloud iam service-accounts create "${SA_NAME}" \
  --display-name "MaxCompute Migration Service" \
  --project "${GCP_PROJECT}"

gcloud projects add-iam-policy-binding "${GCP_PROJECT}" \
  --member "serviceAccount:${SA_EMAIL}" \
  --role "roles/bigquery.dataViewer"

gcloud projects add-iam-policy-binding "${GCP_PROJECT}" \
  --member "serviceAccount:${SA_EMAIL}" \
  --role "roles/bigquery.jobUser"

gcloud iam service-accounts keys create ./mms-sa-key.json \
  --iam-account "${SA_EMAIL}"

Upload to MMS
In the MMS console, go to Data Sources > New > BigQuery. Upload the mms-sa-key.json file and specify the GCP project ID. MMS validates the key and lists the accessible datasets.
Security tip
The JSON key file is retained only during the upload process. After MMS confirms data source registration, it immediately deletes its server-side copy. Rotate the key after the migration is complete.

Create a migration job in MMS

The MMS job wizard guides you through five stages:

Stage 1: Job basic information

Enter a job name and description. Select the registered BigQuery data source and the target MaxCompute project.
Select a job type: Full migration (all rows) or Incremental migration (rows after the checkpoint).

Stage 2: Migration scope

All datasets
Migrates all datasets that the service account can access.
Specific datasets
Migrates one or more datasets, including all tables in those datasets.
Specific tables
Migrates a defined list of tables.
For each option, you can add a partition filter, such as ss_sold_date >= '2026-01-01', to migrate only matching data.

Stage 3: Target configuration

Specify the target MaxCompute project and schema.
Target table format:
- Append Delta Table (recommended for most BigQuery migrations).
- Standard table (used less often for one-time snapshot scenarios).
Naming rule:
Use the {source_dataset}_{source_table} format, or specify a custom prefix and suffix pattern.
Schema strategy:
- Auto-generate
  MMS applies the type mappings in Document 3. This is the most common option.
- Use Existing
  The target table must already exist. MMS validates that the schemas match.

Stage 4: Partition mapping

MMS automatically detects source partitions and provides recommendations:

BigQuery partition	MMS suggested target
Time column (`col DATE`)	`AUTO PARTITIONED BY (TRUNC_TIME(col, 'day') AS <col>_dt)`
Ingestion time (`_PARTITIONTIME`)	`AUTO PARTITIONED BY (TRUNC_TIME(_partitiontime, 'day') AS event_dt)` + `ingestion_time_partition = true`
Integer range	Range clustering (see Document 3 §3.2.4) or a manually defined partition column.
No partition	No partition

Accept the recommendation, or override the granularity or derived column name.

Stage 5: Performance and scheduling

Table concurrency (default: 4, max: 32).
Threads per table (default: 8, max: 32).
Bandwidth limit (unlimited by default; set a limit based on your BigQuery egress budget or shared network constraints).
Schedule the job to run immediately, once at a future time, or periodically for an incremental job.

Click Submit Job. The job appears in the job list with a status of Pending. When it starts, the status changes to Running.

Monitor and troubleshoot

Job progress dashboard
- Overall status: Pending, Running, Completed, Partially-Failed, or Failed.
- Per-table breakdown: Shows the status, rows migrated, bytes migrated, and elapsed time for each table.
- Real-time throughput chart (rows/sec and MB/sec): Use this chart to identify network or quota bottlenecks.
Row count validation: After each table migration completes, MMS runs SELECT COUNT(*) on the source and destination and records whether the counts match. If the counts do not match, the status is marked as failed and the reported difference identifies the mismatch.

Common failure patterns

Symptom	Root cause	Resolution
`Permission denied` (while reading a table)	The service account does not have the `bigquery.dataViewer` role at the dataset level.	Grant the role at the dataset level and retry the failed tables.
`Storage Read API not enabled`	The BigQuery Storage Read API is not enabled for the GCP project.	Enable it by running `gcloud services enable bigquerystorage.googleapis.com`.
Network timeout for large tables	Cross-region egress traffic is constrained by NAT or firewall limits.	Reduce the threads per table, set a bandwidth limit, and enable retries.
Schema mismatch (target already exists)	The target table already exists and its columns differ from the source schema.	Delete the target table or switch to Auto-generate mode.
Row count mismatch	The source table is being written to during migration.	Freeze writes to the source table, or run an incremental migration job after the initial migration to catch up.

Retry behavior
MMS retries at the table level, so partial failures do not rerun tables that completed successfully. To retry failed tables, open Job Details and click Retry Failed Tables.
Log entry point
Detailed per-table logs, including the Tunnel session ID, bytes per session, and retry count, are available on Job Details. When submitting a ticket, include the job ID and table name.