Migrate data from ApsaraDB for Tair (compatible with Redis) to Message Queue for Apache Kafka

Source database

• The server hosting the source database must have sufficient outbound bandwidth. Insufficient bandwidth reduces migration speed.

• For full data migration only: do not write new data to the source instance during migration. To maintain real-time data consistency, select both full data migration and incremental data migration.

• DTS inserts a key with the prefix DTS_REDIS_TIMESTAMP_HEARTBEAT into the source database to record the update timestamp. For cluster architectures, DTS inserts this key into each shard. The key is filtered out during migration and expires after the task ends.

• If the source database is read-only or the DTS account lacks SETEX permissions, the reported migration latency may be inaccurate.

• Increase the repl-backlog-size value in the redis.conf file to improve DTS instance stability.

• If keys in the source database use an expiration policy, the destination may have fewer keys than the source because some keys may expire before they are deleted. Keys without an expiration policy or that have not yet expired are consistent between source and destination.

• If a Tair (Redis OSS-compatible) instance has Transport Layer Security (TLS) encryption enabled, connect to DTS using an SSL-encrypted connection. TLSv1.3 is not supported, and you cannot connect a TLS-enabled Tair (Redis OSS-compatible) instance to DTS as an Alibaba Cloud Instance.

• If Transparent Data Encryption (TDE) is enabled on the source instance, you cannot migrate data using DTS.

• If the source is a Tair (Enterprise Edition) instance using Persistent Memory as the Storage Medium, set the appendonly parameter to yes. For details, see Procedure.

• For self-managed Redis databases:

  • If the database endpoint changes during migration, reconfigure the task.

  • Run config set repl-timeout 600 to set the replication timeout to 600 seconds. Increase this value for large datasets.

  • Public IP Address access with Cluster instance mode is not supported.

• The following events may trigger a full re-migration and cause data inconsistency: a transient connection failure that prevents resumable transmission, a primary/secondary switchover or failover, or an endpoint change on the source Redis instance.

• If the source or destination is a self-managed Redis instance whose endpoint changes during migration (for example, due to instance migration or failover), the migration task may retry, fall behind, or fail. Check the task status and reconfigure the task if needed.

• If the source is a child instance of a Tair (Enterprise Edition) Global Distributed Cache instance, DTS migrates data only from the selected child instance.

Other limits

• If you scale the destination Kafka instance (for example, by adding or removing broker nodes) while the DTS instance is running, restart the DTS instance afterward.

• Make sure DTS can reach both the source and destination. Check that DTS IP addresses are not blocked by security group rules, the bind parameter in redis.conf (for self-managed Redis), or the listeners and advertised.listeners parameters in server.properties (for self-managed Kafka). For details, see Add DTS server IP addresses to a whitelist.

• Full migration consumes read and write resources on both source and destination. Run the DTS instance during off-peak hours when CPU load is below 30%.

• If non-DTS data is written to the destination while the DTS instance is running, data inconsistency may occur and may cause the task to fail.

• For Lua scripts called via EVAL or EVALSHA, DTS cannot confirm successful execution during incremental migration because the destination does not return an explicit result.

• For lists, DTS does not flush existing destination data when using psync or sync. This may result in duplicate data.

• DTS attempts to recover failed instances for up to seven days. End or release the migration instance before switching your business to the destination to prevent automatic recovery from overwriting destination data.

• Restarting a migration instance that includes both full and incremental tasks may cause DTS to re-run both tasks.

• If an instance fails, the DTS helpdesk attempts recovery within 8 hours. During recovery, DTS may restart the instance or adjust instance parameters (database parameters are not modified). For parameters that may be adjusted, see Modify instance parameters.

Step 1: Go to the Data Migration page

Use one of the following methods:

DTS console

1. Log on to the DTS console.

2. In the left-side navigation pane, click Data Migration.

3. In the upper-left corner, select the region where the data migration instance resides.

DMS console

The actual steps may vary based on your DMS console mode and layout. See Simple mode and Customize the layout and style of the DMS console.

1. Log on to the DMS console.

2. In the top navigation bar, move the pointer over Data + AI > DTS (DTS) > Data Migration.

3. From the drop-down list to the right of Data Migration Tasks, select the region where the data migration instance resides.

Step 2: Configure source and destination databases

1. Click Create Task.

2. On the task configuration page, configure the following parameters. Task name Task Name — DTS generates a name automatically. Specify a descriptive name to identify the task. The name does not need to be unique. Source database Select Existing Connection — If you have a database instance registered with DTS, select it from the drop-down list. DTS populates the remaining parameters automatically. See Manage database connections. In the DMS console, select the instance from the Select a DMS database instance drop-down list. If the instance is not registered or you prefer to configure it manually, fill in the parameters below. Database Type — Select Tair/Redis. Access Method — Select Alibaba Cloud Instance. Instance Region — Select the region where the source ApsaraDB for Tair (compatible with Redis) instance resides. Replicate Data Across Alibaba Cloud Accounts — Select No (this guide covers migration between instances under the same Alibaba Cloud account). Instance ID — Select the ID of the source ApsaraDB for Tair (compatible with Redis) instance. Authentication Method — Select an authentication method based on your requirements. This example uses Password Login. Database Password — Enter the password for the database account used to connect to the source instance. The account must have read permission.

   • Default account: enter the password directly.

   • Custom account: enter the password in <custom_account>:<password> format. For example, if the username is admin and the password is Rp829dlwa, enter admin:Rp829dlwa.

   Encryption — Select Non-encrypted or SSL-encrypted based on your requirements. If the access method is not Alibaba Cloud Instance and you select SSL-encrypted, upload a CA Certificate and enter a CA Key. Destination database Select Existing Connection — If you have a database instance registered with DTS, select it from the drop-down list. DTS populates the remaining parameters automatically. See Manage database connections. In the DMS console, select the instance from the Select a DMS database instance drop-down list. If the instance is not registered or you prefer to configure it manually, fill in the parameters below. Database Type — Select Kafka. Access Method — Select Alibaba Cloud Instance. Instance Region — Select the region where the destination Kafka instance resides. Kafka Instance ID — Select the ID of the destination Kafka instance. Encryption — Select Non-encrypted or SCRAM-SHA-256 based on your security requirements. Topic — Select the topic to receive migrated data. Use Kafka Schema Registry — Kafka Schema Registry is a metadata service that provides a RESTful interface for storing and retrieving Avro schemas.

   • No: skip Schema Registry.

   • Yes: enter the URL or IP address used to register the Avro schema in Kafka Schema Registry.

   - Only Redis databases version 6.0 or later support Account + Password Login. - For Secret-free login, enable the password-free access feature on the Redis instance first. See Enable password-free access.

3. Click Test Connectivity and Proceed.

   - Make sure that the CIDR blocks of DTS servers can be automatically or manually added to the security settings of the source and destination databases to allow access from DTS servers. For more information, see Add DTS server IP addresses to a whitelist. - If the source or destination is a self-managed database (access method is not Alibaba Cloud Instance), click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Step 3: Configure objects to be migrated

1. On the Configure Objects page, set the following parameters. Migration Types — Select Full Data Migration and Incremental Data Migration (recommended). If the account configuring the DTS instance lacks SYNC/PSYNC permission on the source, select Full Data Migration only. Processing Mode of Conflicting Tables — Keep the default value. Data Format in Kafka — The format in which data is written to the destination Kafka instance.

   • DTS Avro: data is parsed based on the DTS Avro schema definition. For the full schema definition and field reference, see GitHub.

   • Canal JSON: data is parsed into Canal JSON format. For field descriptions and sample messages, see the Canal JSON section in the data formats topic.

   Kafka Data Compression Format — The compression format for data written to Kafka. Options: Policy for shipping data to Kafka partitions — Select a partitioning policy based on your requirements. Message acknowledgement mechanism — Select a message acknowledgment mechanism based on your requirements. Topic That Stores DDL Information — Select a topic to store DDL information. If left blank, DDL information is stored in the data-receiving topic by default. Source Objects — Select one or more objects from the Source Objects section. Click the icon to move them to the Selected Objects section. To migrate specific keys in a database, use the data filtering feature after adding the database. Selected Objects — To set the topic name, partition count, and additional columns for each source database (DB 0 to DB 255) in the destination Kafka instance, right-click the migration object and configure the mapping. See Configure mapping information.

   • LZ4 (default): low compression ratio, high compression speed.

   • GZIP: high compression ratio, low compression speed. GZIP consumes significant CPU resources.

   • Snappy: medium compression ratio and speed.

2. Click Next: Advanced Settings and configure the following parameters. Dedicated Cluster for Task Scheduling — DTS uses the shared cluster by default. For higher stability, purchase a dedicated cluster. See What is a DTS dedicated cluster. Retry Time for Failed Connections — The time window within which DTS retries failed connections after the task starts. Valid values: 10–1,440 minutes. Default: 720 minutes. Set this to at least 30 minutes. If DTS reconnects within the retry window, the task resumes; otherwise, the task fails. Retry Time for Other Issues — The time window for retrying failed DDL or DML operations. Valid values: 1–1,440 minutes. Default: 10 minutes. Set this to at least 10 minutes. This value must be smaller than Retry Time for Failed Connections. Enable Throttling for Full Data Migration — Limits resource usage during full migration. When enabled, configure Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s) to reduce load on the destination. Available only when Full Data Migration is selected. Enable Throttling for Incremental Data Migration — Limits resource usage during incremental migration. When enabled, configure RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s). Available only when Incremental Data Migration is selected. Use Slave Node — For self-managed Redis databases in Cluster instance mode, choose whether DTS reads from the primary or secondary node. Default: No (reads from the primary node). Environment Tag — Optionally tag the instance to identify its environment (for example, production or test). Configure ETL — Enable the extract, transform, and load (ETL) feature to transform data during migration. See What is ETL? and Configure ETL in a data migration or data synchronization task. Monitoring and Alerting — Configure alerts to notify contacts when the task fails or migration latency exceeds the threshold. See Configure monitoring and alerting.

   - When multiple tasks share the same source or destination database with different retry windows, the most recently set value takes precedence. - DTS charges for the instance during retries. Set the retry window to match your business requirements and release the DTS instance promptly after the source and destination are no longer needed.

Step 4: Run the precheck

Click Next: Save Task Settings and Precheck.

To preview the OpenAPI parameters for this task, move the pointer over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters before proceeding.

DTS runs a precheck before the migration task can start. The task starts only after the precheck passes.

If the precheck fails, click View Details next to each failed item, resolve the issue, then click Precheck Again.

If an alert is triggered: if the alert can be ignored, click Confirm Alert Details, click Ignore in the View Details dialog box, confirm, then click Precheck Again. Note that ignoring an alert may result in data inconsistency.

Step 5: Purchase and start the instance

1. Wait until the Success Rate reaches 100%, then click Next: Purchase Instance.

2. On the Purchase Instance page, configure the instance class. Resource Group — The resource group for the migration instance. Default: default resource group. See What is Resource Management? Instance Class — Select an instance class based on the required migration speed. See Instance classes of data migration instances.

3. Read and select the Data Transmission Service (Pay-as-you-go) Service Terms check box.

4. Click Purchase And Start, then click OK.

You can monitor progress on the data migration page.

If you selected both Full Data Migration and Incremental Data Migration, the tasks are merged and displayed as Incremental Data Migration on the task list page.