Use Data Transmission Service (DTS) to migrate data from a PolarDB-X 2.0 instance to the LindormTable engine of a Lindorm instance. This guide covers both full data migration (one-time cutover) and full + incremental data migration (continuous sync for zero-downtime cutover).
In this guide, you will:
-
Verify prerequisites and required account permissions
-
Configure source and destination database connections
-
Select migration types and objects
-
Configure advanced settings and alerting
-
Run the precheck and launch the migration task
Prerequisites
Before you begin, make sure you have:
-
A Lindorm instance that uses LindormTable as its database engine, with storage capacity exceeding the data volume to migrate. See Create an instance.
-
The MySQL-compatible connection endpoint enabled on the Lindorm instance. See Enable the MySQL-compatible feature.
-
A database (namespace) and a wide table created in the Lindorm instance. Pre-partition the wide table based on the full data if needed. See Connect to and use LindormTable using the MySQL command line, Connect to and use LindormTable using Lindorm-cli, Access LindormTable using Lindorm Shell, CREATE TABLE, and Data type mapping.
All objects you create must meet the requirements in Quotas and limits.
Permissions for database accounts
Grant the following permissions to the accounts used by DTS.
| Database | Schema migration | Full migration | Incremental migration |
|---|---|---|---|
| Source PolarDB-X 2.0 | SELECT |
SELECT |
REPLICATION SLAVE, REPLICATION CLIENT, and SELECT on migrated objects |
| Destination Lindorm | — | Read and write on the destination namespace | Read and write on the destination namespace |
To create accounts and grant permissions:
-
For PolarDB-X 2.0: see Manage database accounts and Account permission issues during data synchronization.
-
For Lindorm: see User management.
Example: Grant incremental migration permissions on PolarDB-X 2.0
-- Grant replication and select permissions (incremental migration)
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO '<dts_user>'@'%';
GRANT SELECT ON <db_name>.* TO '<dts_user>'@'%';
FLUSH PRIVILEGES;Replace <dts_user> with the DTS account name and <db_name> with the database to migrate.
Billing
| Migration type | Link configuration fee | Data transfer cost |
|---|---|---|
| Full data migration | Free | Free, unless the Access Method of the destination is Public IP Address. See Billable items. |
| Incremental data migration | Paid. See Billing overview. | — |
Limitations
Source database limitations
| Limitation | Applies to |
|---|---|
| The server to which the source database belongs must have sufficient outbound bandwidth. Otherwise, the data migration speed is affected. | Full + incremental |
| Read-only instances of Enterprise Edition PolarDB-X 2.0 are not supported. | Full + incremental |
| Schema migration is not supported. | Full + incremental |
BIT type data cannot be migrated. |
Full + incremental |
| Tables with uppercase letters in their names cannot be migrated. | Full + incremental |
TABLEGROUP and databases or schemas with the Locality attribute are not supported. |
Full + incremental |
Tables whose names are reserved words (for example, select) cannot be migrated. |
Full + incremental |
| The migrated data must contain at least one non-primary key field. Migrating only primary key fields is not supported. | Full + incremental |
Tables must have PRIMARY KEY or UNIQUE constraints with all fields unique. Otherwise, the destination may contain duplicate records. |
Full + incremental |
| When using object name mapping (renaming tables or columns), a single task supports up to 1,000 tables. For more than 1,000 tables, split the migration across multiple tasks or configure a task to migrate the entire database instead. | Full + incremental |
Set the binlog_row_image parameter to full. Binary logging is enabled by default on PolarDB-X 2.0 instances. See Parameter Settings. |
Incremental only |
| Do not perform DDL operations during migration. DTS queries the source database and creates metadata locks, which may block DDL operations. | Full only |
| Do not write to the source database during migration. For continuous consistency, select both full data migration and incremental data migration. | Full only |
| If the network configuration of the PolarDB-X 2.0 instance changes, the migration task may experience latency for a period of time. | Full + incremental |
Other limitations
| Limitation | Applies to |
|---|---|
| DTS can only write data to the LindormTable engine of a Lindorm instance. | Full + incremental |
Empty strings of the VARBINARY type in the source are treated as null values in DTS and Lindorm. |
Full + incremental |
If DECIMAL fields in the source and destination have different precision, the task fails. Make sure the precision matches. |
Full + incremental |
| Migrated data must meet the requirements in Limits on data requests. Otherwise, the task fails. | Full + incremental |
DTS periodically executes CREATE DATABASE IF NOT EXISTS \test\`` on the source to advance the binary log offset. If the Whether to delete SQL operations on heartbeat tables of forward and reverse tasks option is set to Yes (or the source account lacks permission to create databases), and no DML operations occur on the source for an extended period, migration latency reporting may become inaccurate. |
Incremental only |
| Run migration during off-peak hours. Full data migration consumes read and write resources on both the source and destination. Migrate when the CPU load on both databases is below 30%. | Full only |
Full data migration uses concurrent INSERT operations, which cause table fragmentation. After migration, destination storage usage will be larger than the source. |
Full only |
| If data is written to the destination from sources other than DTS during migration, data inconsistency may occur and the migration task may fail. | Full + incremental |
| If a DTS instance fails, DTS support will try to recover it within 8 hours. During recovery, the instance may be restarted or have its parameters adjusted. Only DTS instance parameters are modified — source and destination database parameters are not changed. See Modify instance parameters. | Full + incremental |
If the migration instance shows a long latency, perform a DML operation on the source database to update the latency display.
SQL operations supported for incremental migration
| Operation type | SQL statement |
|---|---|
| DML | INSERT, UPDATE, DELETE |
| DDL | CREATE TABLE, DROP TABLE, ADD COLUMN |
Notes on DDL support:
-
For PolarDB-X 2.0 Enterprise Edition,
CREATE TABLEis not supported in incremental migration. -
For
ADD COLUMN, additional attributes are dropped. For example, if the source statement isALTER TABLE test ADD COLUMN col INT DEFAULT 0;, DTS executesALTER TABLE test ADD COLUMN col INT;on the destination. This is a LindormTable limitation.
Create a migration task
Step 1: Go to the Data Migration page
Use either console:
DTS console
-
Log on to the DTS console.
-
In the left-side navigation pane, click Data Migration.
-
In the upper-left corner, select the region where the migration instance resides.
DMS console
The exact navigation path may vary by DMS console mode and layout. See Simple mode and Customize the layout and style of the DMS console.
-
Log on to the DMS console.
-
In the top navigation bar, move the pointer over Data + AI > DTS (DTS) > Data Migration.
-
From the drop-down list to the right of Data Migration Tasks, select the region where the migration instance resides.
Step 2: Configure source and destination databases
-
Click Create Task.
-
Configure the parameters in the following table.
| Category | Parameter | Description |
|---|---|---|
| — | Task Name | An informative name to identify the task. Task names do not need to be unique. DTS auto-generates a name if you leave this blank. |
| Source Database | Select Existing Connection | If you have a database instance registered with DTS, select it from the drop-down list. DTS auto-populates the connection parameters. Otherwise, configure the parameters below manually. In the DMS console, select the instance from Select a DMS database instance. See Manage database connections. |
| Database Type | In the Relational Database section, select PolarDB-X 2.0. | |
| Access Method | Select Alibaba Cloud Instance. | |
| Instance Region | Select the region where the source PolarDB-X 2.0 instance resides. | |
| Replicate Data Across Alibaba Cloud Accounts | Select No for same-account migration. | |
| Instance ID | Select the ID of the source PolarDB-X 2.0 instance. | |
| Database Account | Enter the database account. See Permissions for database accounts. | |
| Database Password | Enter the password for the database account. | |
| Destination Database | Select Existing Connection | If you have a database instance registered with DTS, select it from the drop-down list. Otherwise, configure the parameters below manually. In the DMS console, select the instance from Select a DMS database instance. See Manage database connections. |
| Database Type | In the NoSQL Database section, select Lindorm. | |
| Access Method | Select Alibaba Cloud Instance. | |
| Instance Region | Select the region where the destination Lindorm instance resides. | |
| Instance ID | Select the ID of the destination Lindorm instance. | |
| Database Account | Enter the database account. See Permissions for database accounts. | |
| Database Password | Enter the password for the database account. |
-
Click Test Connectivity and Proceed.
Make sure DTS server CIDR blocks are added to the whitelist of both the source and destination databases. See Add DTS server IP addresses to a whitelist.
Step 3: Configure objects to migrate
On the Configure Objects page, set the following parameters.
| Parameter | Description |
|---|---|
| Migration Types | Select Full Data Migration for a one-time migration of existing data. Select both Full Data Migration and Incremental Data Migration to keep source and destination in sync during migration and ensure service continuity. Note
If you select only full data migration, do not write to the source database during the task. |
| Processing Mode of Conflicting Tables | Keep the default value. |
| Capitalization of Object Names in Destination Instance | Keep the default value. |
| Source Objects | Select one or more objects at the database or table level, then click the arrow icon to add them to Selected Objects. |
| Selected Objects | If the database (namespace), table, or column names in the destination differ from the source, use object name mapping. See Map table and column names. Note
Renaming an object may cause dependent objects to fail migration. To filter rows, right-click a table in Selected Objects and specify WHERE conditions. See Specify filter conditions. To select specific SQL operations for incremental migration, right-click an object in Selected Objects and select the operations. |
Click Next: Advanced Settings.
Step 4: Configure advanced settings
| Parameter | Description |
|---|---|
| Dedicated Cluster for Task Scheduling | By default, DTS schedules tasks to the shared cluster. For higher stability, purchase a dedicated cluster. See What is a DTS dedicated cluster. |
| Retry Time for Failed Connections | How long DTS retries after a connection failure. Valid values: 10–1,440 minutes. Default: 720 minutes. Set to more than 30 minutes. If DTS reconnects within the retry period, the task resumes; otherwise, it fails. Note
When multiple tasks share a source or destination database, the most recently set retry time takes precedence. You are charged for the DTS instance during retry. We recommend that you specify the retry time range based on your business requirements. You can also release the DTS instance at the earliest opportunity after the source database and destination instance are released. |
| Retry Time for Other Issues | How long DTS retries after DDL or DML failures. Valid values: 1–1,440 minutes. Default: 10 minutes. Set to more than 10 minutes. Must be smaller than Retry Time for Failed Connections. |
| Enable Throttling for Full Data Migration | Limits read/write load on source and destination during full migration. Configure Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s). Available only when Full Data Migration is selected. |
| Enable Throttling for Incremental Data Migration | Limits load during incremental migration. Configure RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s). Available only when Incremental Data Migration is selected. |
| Whether to delete SQL operations on heartbeat tables of forward and reverse tasks | Controls whether DTS writes heartbeat SQL to the source database. Yes: no heartbeat writes; migration latency display may be inaccurate. No: heartbeat writes enabled; may affect physical backup and cloning of the source database. |
| Environment Tag | (Optional) A tag to identify the environment. |
| Configure ETL | Whether to enable the extract, transform, and load (ETL) feature. Yes: enter data processing statements in the code editor. See Configure ETL in a data migration or data synchronization task. No: ETL is not used. See What is ETL? |
| Monitoring and Alerting | Whether to configure alerting. Yes: set the alert threshold and notification contacts; contacts are notified if the task fails or latency exceeds the threshold. See Configure monitoring and alerting. No: no alerting. |
Step 5: Run the precheck
-
Click Next: Save Task Settings and Precheck. To preview the API parameters for this task, hover over the button and click Preview OpenAPI parameters before proceeding.
-
Wait for the precheck to complete.
The task cannot start until it passes the precheck. If any item fails, click View Details next to the failed item, resolve the issue, and run the precheck again. If an item triggers an alert that can be safely ignored, click Confirm Alert Details, then Ignore, then OK, and run the precheck again. Ignoring alerts may cause data inconsistency.
Step 6: Purchase the instance and start the task
-
After Success Rate reaches 100%, click Next: Purchase Instance.
-
On the Purchase Instance page, configure the following parameters.
| Section | Parameter | Description |
|---|---|---|
| New Instance Class | Resource Group | The resource group for the migration instance. Default: default resource group. See What is Resource Management? |
| Instance Class | The instance class determines migration speed. Select based on your data volume and business requirements. See Instance classes of data migration instances. |
-
Read and select the Data Transmission Service (Pay-as-you-go) Service Terms check box.
-
Click Buy and Start, then click OK in the confirmation dialog.
View task progress on the Data Migration page.
-
Full data migration only: the task stops automatically when complete. Status shows Completed.
-
Full + incremental data migration: the task runs continuously. Status shows Running.
Data type mapping
The following table shows how PolarDB-X 2.0 source types map to Lindorm types.
| Source data type | Lindorm data type | Notes |
|---|---|---|
BOOLEAN |
BOOLEAN |
— |
BIT |
BOOLEAN |
— |
TINYINT |
TINYINT |
— |
SMALLINT |
SMALLINT |
— |
INTEGER |
INTEGER |
— |
BIGINT |
BIGINT |
— |
BIGINT UNSIGNED |
BIGINT |
Only values within the signed BIGINT range (−9,223,372,036,854,775,808 to 9,223,372,036,854,775,807) are supported. |
FLOAT |
FLOAT |
— |
DOUBLE |
DOUBLE |
— |
DECIMAL |
DECIMAL |
Precision must match the source field exactly. The task fails if precision differs. |
CHAR / VARCHAR / TEXT / TINYTEXT / MEDIUMTEXT / LONGTEXT |
CHAR / VARCHAR |
— |
BINARY |
BINARY |
— |
BLOB |
VARBINARY |
— |
VARBINARY |
VARBINARY |
Empty strings are treated as null values. |
TIMESTAMP |
TIMESTAMP |
— |
YEAR |
INTEGER |
— |
DATE |
DATE (Lindorm 2.8.0.2 and later) / VARCHAR (earlier versions) |
— |
DATETIME |
VARCHAR |
Map to VARCHAR, not TIMESTAMP. If you map it to TIMESTAMP, data inconsistency may occur due to time zone differences. Use the ETL feature during task configuration to ensure data consistency. |
TIME |
TIME (Lindorm 2.8.0.2 and later) / VARCHAR (earlier versions) |
For version 2.8.0.2 and later: format is hh:mm:ss. Data exceeding this format is truncated (for example, 08:11:15.354 becomes 08:11:15). |
JSON |
JSON |
— |