Migrate data from RDS for MySQL to Lindorm

Source database limitations

• The server that hosts the source database must have sufficient outbound bandwidth. Otherwise, the data migration speed is affected.

• Tables to be migrated must have primary keys and unique fields. Without them, duplicate data may appear in the destination.

• If you migrate at the table level and edit objects (such as mapping table or column names), a single task supports a maximum of 1,000 tables. If you exceed this limit, split the tables into multiple tasks or configure the task to migrate the entire database.

• If the source is MySQL 8.0.23 or later and the data includes invisible columns, DTS cannot read those columns and data loss occurs. To make invisible columns visible before migrating, run:

  ALTER TABLE <table_name> ALTER COLUMN <column_name> SET VISIBLE;

  For more information, see Invisible Columns. Tables without explicit primary keys automatically generate invisible primary keys — make those visible too. See Generated Invisible Primary Keys.

Incremental migration only:

• Binary logging must be enabled with the following settings. Without these, the precheck fails and the task cannot start. Binary log retention requirements: If binary logs are deleted before DTS can read them, the task may fail or data inconsistency may occur, and the DTS Service Level Agreement (SLA) may not apply.

  • ApsaraDB RDS for MySQL: at least 3 days (7 days recommended). See Configure parameters based on which the system automatically deletes the binary log files of an RDS instance.

  • Self-managed MySQL: at least 7 days.

  Parameter	Required value	Effect if not set
  binlog_format	ROW	Setting to STATEMENT or MIXED can cause data inconsistency in the destination.
  binlog_row_image	full	Partial row images may cause incomplete data replication.
  log_slave_updates	ON	Required only for self-managed MySQL in a dual-primary cluster. Without it, DTS cannot obtain all binary logs.

Full migration only:

• Do not perform DDL operations on the source during full migration. DDL operations may cause the task to fail.

  During full migration, DTS queries the source database and creates metadata locks, which can block DDL operations on the source.

• Do not write new data to the source if you select only Full Data Migration. To keep source and destination in sync throughout migration, select both Full Data Migration and Incremental Data Migration.

• Full data migration cannot be performed if the source RDS for MySQL instance has the EncDB feature enabled.

  RDS for MySQL instances with Transparent Data Encryption (TDE) enabled support schema migration, full data migration, and incremental data migration.

Other limitations

• Schema migration to Lindorm relies on Lindorm's own capabilities for schema creation.

• Lindorm has a default column count limit for indexes on non-primary key columns. DTS cannot create such indexes during schema migration when the column count exceeds this limit. To use such indexes, create them manually. For details, see ALTER SYSTEM.

• Migration of INDEX, PARTITION, VIEW, PROCEDURE, FUNCTION, TRIGGER, and foreign key objects is not supported.

• Migration of BIT data type is not supported.

• You cannot migrate only primary key fields. The data must contain at least one non-primary key field.

• Data can only be written to the wide table engine of a Lindorm instance.

• Empty strings of the VARBINARY type are treated as null by both DTS and Lindorm.

• If the precision of a DECIMAL field in the destination differs from the source, the task fails.

• Data written to Lindorm must comply with limits on data requests. Requests exceeding these limits cause the task to fail.

• Full migration runs concurrent INSERT operations, which causes tablespace fragmentation in the destination. The destination tablespace will be larger than the source after full initialization completes.

• Do not run online DDL tools such as pt-online-schema-change on migration objects in the source during migration. This causes the migration to fail.

• If data is written to the destination by sources other than DTS during migration, data inconsistency may occur.

• Data generated by binary log change operations — such as data restored from a physical backup or cascade operations — is not migrated to the destination.

  If such data is not migrated, you can run a full migration again as long as your business is not affected.

• If a DTS instance fails, the DTS helpdesk attempts to recover it within 8 hours. Recovery operations may include restarting the instance and adjusting DTS instance parameters (not database parameters). For parameters that may be adjusted, see Modify instance parameters.

Special cases

Self-managed MySQL source:

• If a primary/secondary switchover occurs on the source while the migration task is running, the task fails.

• DTS periodically runs CREATE DATABASE IF NOT EXISTS `test` on the source to advance the binary log file position.

• If no DML operations are performed on the source for an extended period, migration latency readings may be inaccurate. Run a DML operation on the source to refresh the latency. If you select an entire database as the migration object, create a heartbeat table that receives updates every second.

ApsaraDB RDS for MySQL source:

• Incremental migration only: A read-only RDS for MySQL V5.6 instance cannot be used as the source because it does not record transaction logs.

• DTS periodically runs CREATE DATABASE IF NOT EXISTS `test` on the source to advance the binary log file position.

Step 1: Open the Data Migration page

Use one of the following methods:

Step 2: Configure source and destination databases

1. Click Create Task.

2. Configure the source and destination databases using the following parameters:

   Source database:

   Parameter	Description
   Task Name	The name of the DTS task. DTS generates a name automatically. Specify a descriptive name to make the task easy to identify. The name does not need to be unique.
   Select Existing Connection	If the source instance is already registered with DTS, select it from the drop-down list — DTS populates the database parameters automatically. Otherwise, configure the parameters below manually.
   Database Type	Select MySQL.
   Access Method	Select Alibaba Cloud Instance.
   Instance Region	Select the region where the source RDS for MySQL instance resides.
   Replicate Data Across Alibaba Cloud Accounts	Select No if the source is in the same Alibaba Cloud account.
   RDS Instance ID	Select the ID of the source RDS for MySQL instance.
   Database Account	Enter the database account for the source instance. For required permissions, see Database account permissions.
   Database Password	Enter the password for the database account.
   Encryption	Select Non-encrypted or SSL-encrypted. To use SSL encryption, enable it on the RDS for MySQL instance first. See Use a cloud certificate to enable SSL encryption.

   Destination database:

   Parameter	Description
   Select Existing Connection	If the destination instance is already registered with DTS, select it from the drop-down list. Otherwise, configure the parameters below manually.
   Database Type	Select Lindorm.
   Access Method	Select Alibaba Cloud Instance.
   Instance Region	Select the region where the destination Lindorm instance resides.
   Instance ID	Select the ID of the destination Lindorm instance.
   Database Account	Enter the database account for the destination instance.
   Database Password	Enter the password for the database account.

3. Click Test Connectivity and Proceed.

   Make sure that the CIDR blocks of DTS servers are added to the security settings of the source and destination databases. See Add DTS server IP addresses to a whitelist. If the source or destination is a self-managed database not accessed via Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Step 3: Configure objects to migrate

1. On the Configure Objects page, set the following parameters:

   If you do not select Incremental Data Migration, do not write new data to the source during migration to ensure data consistency.

   Parameter	Description
   Migration Types	Select Full Data Migration for a one-time migration. Select both Full Data Migration and Incremental Data Migration to keep the destination in sync with the source throughout the migration.
   Processing Mode of Conflicting Tables	Keep the default value.
   Capitalization of Object Names in Destination Instance	Controls the capitalization of database names, table names, and column names in the destination. The default is DTS default policy. For more information, see Specify the capitalization of object names in the destination instance.
   Source Objects	Select the databases or tables to migrate, then click to move them to Selected Objects.
   Selected Objects	If database, table, or column names in the destination Lindorm instance differ from those in the source, use object name mapping. For details, see Map object names. To apply row-level filters, right-click a table and specify WHERE conditions. See Specify filter conditions. To select which SQL operations to include in incremental migration, right-click an object and select the operations.

2. Click Next: Advanced Settings and configure the following:

   Parameter	Description
   Dedicated Cluster for Task Scheduling	By default, DTS uses the shared cluster. For improved stability, purchase a dedicated cluster. See What is a DTS dedicated cluster.
   Retry Time for Failed Connections	How long DTS retries if the source or destination is unavailable. Valid values: 10–1,440 minutes. Default: 720 minutes. Set this to more than 30 minutes. If DTS reconnects within this window, it resumes the task; otherwise, the task fails. Note: if multiple tasks share the same source or destination, the most recently set value applies. You are charged for the DTS instance during retries.
   Retry Time for Other Issues	How long DTS retries failed DDL or DML operations. Valid values: 1–1,440 minutes. Default: 10 minutes. Set this to more than 10 minutes. This value must be smaller than Retry Time for Failed Connections.
   Enable Throttling for Full Data Migration	Throttles full migration to reduce load on source and destination. Configure Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s). Available only when Full Data Migration is selected.
   Enable Throttling for Incremental Data Migration	Throttles incremental migration. Configure RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s). Available only when Incremental Data Migration is selected.
   Environment Tag	Select a tag to identify the instance. No tag is required for this example.
   Whether to delete SQL operations on heartbeat tables of forward and reverse tasks	Controls whether DTS writes heartbeat table SQL operations to the source while the instance runs. Select Yes to skip writing (a latency value may appear on the DTS instance). Select No to write heartbeat operations (this may affect physical backup and cloning of the source).
   Configure ETL	Select Yes to configure extract, transform, and load (ETL) rules in the code editor. See Configure ETL in a data migration or data synchronization task. Select No to skip ETL configuration.
   Monitoring and Alerting	Select Yes to receive alerts when the task fails or migration latency exceeds the threshold. Configure the alert threshold and notification settings. See Configure monitoring and alerting when you create a DTS task.

Step 4: Run the precheck

Click Next: Save Task Settings and Precheck.

To view the OpenAPI parameters for this task configuration, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters before proceeding.

DTS runs a precheck before starting the migration. If the precheck fails:

• Click View Details next to each failed item, resolve the issue, then click Precheck Again.

• If a precheck alert can be ignored, click Confirm Alert Details, then Ignore in the dialog box, then OK, then Precheck Again. Ignoring alerts may result in data inconsistency.

Step 5: Purchase and start the instance

1. Wait until Success Rate reaches 100%, then click Next: Purchase Instance.

2. On the Purchase Instance page, configure the following:

   Parameter	Description
   Resource Group	The resource group for the migration instance. Default: default resource group. See What is Resource Management?
   Instance Class	DTS provides instance classes that vary in the migration speed. You can select an instance class based on your business scenario. See Instance classes of data migration instances.

3. Read and accept the Data Transmission Service (Pay-as-you-go) Service Terms, then click Buy and Start. Click OK in the confirmation message.

You can view the progress of the task on the Data Migration page.

If a data migration task cannot be used to migrate incremental data, the task automatically stops. The Completed status is displayed.

If a data migration task can be used to migrate incremental data, the task does not automatically stop. The incremental data migration task runs continuously. The Running status is displayed.