Migrate from a self-managed Redis to Tair (Redis OSS-Compatible)

Prerequisites

• The version of the self-managed Redis database is 2.8, 3.0, 3.2, 4.0, 5.0, or 6.0.

• The self-managed Redis database is deployed in a standalone architecture. The cluster architecture is not supported.

  Note

  If your database uses a cluster architecture, you can migrate data by using the data synchronization feature. For more information, see Synchronize data from a self-managed Redis cluster to a Tair (Redis OSS-Compatible) instance.

• The psync or sync command can run on the self-managed Redis database.

• The destination Tair (Redis OSS-Compatible) instance must have more storage space than the source self-managed Redis database occupies.

• The repl-timeout parameter in the source Redis instance defines the replication timeout between slave and master nodes, with a default of 60 seconds. We recommend running the config set repl-timeout 600 command to increase the timeout to 600 seconds. If the source database contains a large amount of data, you can increase the value of the repl-timeout parameter as needed.

Usage notes

• DTS consumes resources from both the source and destination databases, which increases the load on the database servers. If your database already has high traffic or low specifications, this added load can lead to service unavailability.

• If the destination database runs out of memory and triggers data eviction, data may become inconsistent between the source and destination. This occurs because the default data eviction policy (maxmemory-policy) for Tair (Redis OSS-Compatible) is volatile-lru. However, this does not affect normal task operation.

  To prevent this issue, we recommend setting the data eviction policy for the destination database to noeviction. With this policy, if the destination runs out of memory, write operations fail, the task stops, and no data is lost to eviction.

  Note

  For more information about data eviction policies, see Data eviction policies in Redis.

• If the source database has keys with an expiration policy, the destination database may report fewer keys when checked with a command like info. This is because expired keys may not be immediately deleted.

  Note

  The number of keys that do not have an expiration policy or have not expired is the same in both the source and destination databases.

• For Lua scripts that are called by using EVAL or EVALSHA, DTS cannot confirm whether the scripts are successfully executed during incremental data migration. This is because the destination database does not explicitly return an execution result.

• For List data types, DTS does not perform a Flush operation on existing data in the destination database when it transfers data by using psync or sync. This may result in duplicate data.

• If the self-managed Redis instance undergoes scaling, such as adding or removing shards, or resizing, such as increasing memory, during the migration, you must reconfigure the task. To ensure data consistency, clear the migrated data from the destination instance before you reconfigure the task.

• If the connection address of the self-managed Redis instance changes during the migration, you must reconfigure the task.

• DTS automatically tries to recover failed migration tasks. When you are ready to switch your business to the destination instance, you must stop or release the task. This prevents the task from being automatically recovered, which could cause data from the source database to overwrite data in the destination instance.

• When migrating from standalone to cluster Redis, note that clusters allow a command to operate on only one slot. Multi-key operations across slots return the following error:

  CROSSSLOT Keys in request don't hash to the same slot

  To avoid task interruption, use only single-key operations.

• If the destination instance is a cluster instance and a shard reaches its memory limit, or if the destination instance runs out of storage space, the DTS task fails due to an out of memory (OOM) error.

• DTS inserts a key prefixed with DTS_REDIS_TIMESTAMP_HEARTBEAT into the source to track update timestamps (one per shard in cluster mode). This key is filtered during migration and expires after the task completes.

• If the source instance is read-only or the DTS account does not have write (SETEX) permissions, the reported latency may be inaccurate.

• Data migration is not supported if the destination instance has Transparent Data Encryption (TDE) enabled.

• The following events during migration may cause full data migration to restart, leading to data inconsistency:

  • A transient connection interruption on the source or target Redis instance causes resumable transmission to fail.

  • A primary/secondary switchover or failover occurs on the source or target Redis instance.

  • The connection address of the source or target Redis instance changes.

• To migrate incremental data, the source database account used by the task must have the PSYNC and SYNC permissions.

• If a Tair (Redis OSS-Compatible) instance has Transport Layer Security (TLS) encryption enabled, you must connect to it by using the SSL-encrypted method in DTS. TLSv1.3 is not supported. You cannot connect to an SSL-enabled Tair (Redis OSS-Compatible) instance by setting the Alibaba Cloud Instance to Alibaba Cloud Instance.

• If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust its parameters.

  Note

  Only DTS task parameters are modified—not database parameters. Parameters that may be adjusted include those listed in Modify instance parameters.

Billing

Migration types

• Full data migration

  DTS migrates all existing data from the migration objects in the self-managed Redis database to the Tair (Redis OSS-Compatible) instance.

  Note

  If you perform only a full data migration, do not write new data to the self-managed Redis database during the migration. This ensures data consistency.

• Incremental data migration

  After the full data migration is complete, DTS synchronizes incremental data changes from the self-managed Redis database to the Tair (Redis OSS-Compatible) instance. This allows for a smooth migration to the cloud without service downtime.

Supported commands for incremental migration

• APPEND

• BITOP, BLPOP, BRPOP, BRPOPLPUSH

• DECR, DECRBY, DEL

• EVAL, EVALSHA, EXEC, EXPIRE, EXPIREAT

• FLUSHALL, FLUSHDB

• GEOADD, GETSET

• HDEL, HINCRBY, HINCRBYFLOAT, HMSET, HSET, HSETNX

• INCR, INCRBY, INCRBYFLOAT

• LINSERT, LPOP, LPUSH, LPUSHX, LREM, LSET, LTRIM

• MOVE, MSET, MSETNX, MULTI

• PERSIST, PEXPIRE, PEXPIREAT, PFADD, PFMERGE, PSETEX, PUBLISH

• RENAME, RENAMENX, RESTORE, RPOP, RPOPLPUSH, RPUSH, RPUSHX

• SADD, SDIFFSTORE, SELECT, SET, SETBIT, SETEX, SETNX, SETRANGE, SINTERSTORE, SMOVE, SPOP, SREM, SUNIONSTORE

• ZADD, ZINCRBY, ZINTERSTORE, ZREM, ZREMRANGEBYLEX, ZUNIONSTORE, ZREMRANGEBYRANK, ZREMRANGEBYSCORE

• XADD, XCLAIM, XDEL, XAUTOCLAIM, XGROUP CREATECONSUMER, XTRIM

Preparations (for incremental migration)

To ensure that the incremental data migration task runs as expected, disable the limit on the replication output buffer. This topic uses a server that runs Linux as an example.

1. Use the redis-cli tool to connect to the self-managed Redis database.

   Note

   You can use redis-cli after you install Redis. For more information, visit the official Redis community website.

   redis-cli -h <host> -p <port> -a <password>

   Note

   • <host>: The connection address of the self-managed Redis database. You can use 127.0.0.1 for a local connection.

   • <port>: The service port of the self-managed Redis database. The default value is 6379.

   • <password>: The password of the self-managed Redis database.

   Example:

   redis-cli -h 127.0.0.1 -p 6379 -a Test123456

2. Run the following command to disable the limit on the replication output buffer.

   config set client-output-buffer-limit 'slave 0 0 0'

Procedure

1. Log on to the Data Transmission Service (DTS) console.

   Note

   If you are automatically redirected to the Data Management (DMS) console, you can click the icon in the lower-right corner and then click to return to the classic DTS console.

2. In the navigation pane on the left, click Data Migration.

3. At the top of the Migration Tasks page, select the region where the destination instance is located.

4. In the upper-right corner of the page, click Create Data Migration Task.

5. Configure the source and destination databases.

   Category	Configuration	Description
   N/A	Task name	DTS automatically generates a task name. For easy identification, specify a meaningful name. The name does not need to be unique.
   Source database	Instance type	Select a value based on the deployment location of the source database. This topic uses User-Created Database with Public IP Address as an example. Note If your self-managed database is a different instance type, you must perform the required preparations. For more information, see Preparation overview.
   	Instance region	If you select User-Created Database with Public IP Address as the instance type, you do not need to set Instance Region. Note If you configured an allow list for your self-managed Redis database, click Get IP Address Segment of DTS next to the Instance Region field to obtain the IP addresses of the DTS servers. Then, add these IP addresses to the allow list of your self-managed Redis database.
   	Database type	Select Redis.
   	Instance mode	This parameter is fixed to Standalone. The cluster mode is not supported.
   	Hostname or IP address	Enter the connection address of the self-managed Redis database. In this case, enter the public IP address.
   	Port	Enter the service port of the self-managed Redis database. The default value is 6379. Note In this case, the service port must be accessible from the Internet.
   	Database password	Enter the password for the self-managed Redis database. Note After you enter the source database information, you can click Test Connectivity next to Database Password to verify that the information is correct. If the information is correct, the message Passed is displayed. If the message Failed is displayed, click Diagnose next to the Failed message and adjust the source database information based on the prompts.
   Destination database	Instance type	Select Redis Instance.
   	Instance region	Select the region of the destination Redis instance.
   	Redis instance ID	Select the ID of the destination Redis instance.
   	Database password	Enter the password for the destination Redis instance. Note After you enter the destination database information, you can click Test Connectivity after Database Password to verify that the entered information is correct. If the information is correct, a Passed message is displayed. If a Failed message is displayed, click Diagnose after Failed and adjust the destination database information based on the prompts.

6. After you complete the configuration, click Set Whitelist and Next in the lower-right corner of the page.

   If the source or destination database is an Alibaba Cloud database instance (such as RDS for MySQL or ApsaraDB for MongoDB), DTS automatically adds the IP addresses of the DTS servers in the corresponding region to the allowlist of the database instance. If the source or destination database is a self-hosted database on an ECS instance, DTS automatically adds the IP addresses of the DTS servers in the corresponding region to the security rules of the ECS instance. You must also ensure that the self-hosted database allows access from the ECS instance. If the database is deployed in a cluster on multiple ECS instances, you must manually add the IP addresses of the DTS servers in the corresponding region to the security rules of each of the other ECS instances. If the source or destination database is a self-hosted database in an on-premises data center (IDC) or a database from another cloud provider, you must manually add the IP addresses of the DTS servers in the corresponding region to allow access from the DTS servers. For the IP addresses of the DTS servers, see IP address ranges of DTS servers.

   Warning

   Adding the public CIDR blocks of DTS servers, whether automatically or manually, may introduce security risks. By using this product, you acknowledge and accept these potential risks. You are responsible for implementing basic security measures, including but not limited to using strong passwords, restricting open ports, using authentication for internal API calls, regularly reviewing and restricting unnecessary network segments, or connecting through private networks such as Express Connect, VPN Gateway, or Smart Access Gateway.

7. Select migration objects and migration types.

   Configuration	Description
   Migration types	To perform only a full migration, select Full Data Migration. For a migration with no downtime, select both Full Data Migration and Incremental Data Migration. Note If you do not select Incremental Data Migration, do not write new data to the self-managed Redis database during the migration to ensure data consistency.
   Migration objects	In the Available box, click the database to be migrated, and then click to move it to the Selected Objects box. Note Migration objects are selected at the database level.
   Object renaming	Renaming objects is not supported.
   Connection failure retry period	By default, DTS retries for 12 hours. You can also customize the retry period. If DTS reconnects to the source and destination databases within the specified period, the migration task automatically resumes. Otherwise, the task fails. Note DTS charges for the task runtime during the connection retry period. Customize the retry period based on your business needs, or release the DTS instance as soon as the source and destination instances are released.

8. After you complete these settings, click Pre-check and Start in the lower-right corner of the page.

   Note

   • DTS performs a precheck before the task starts. The task can start only after it passes the precheck.

   • If the precheck fails, click the icon next to the failed item to view details.

     • Fix the issues based on the details, and then run the precheck again.

     • For warnings that do not require a fix, you can select Ignore or Ignore and Rerun Precheck to run the precheck again.

9. After the task passes the precheck, click Next.

10. On the Confirm Order page, select a Instance Class and select the Data Transmission Service (Pay-As-You-Go) Terms of Service checkbox.

11. Click Buy and Start. The migration task starts.

    • Full data migration

      Do not manually stop the task. Wait for it to complete automatically to avoid data loss.

    • Incremental data migration

      The task does not stop automatically. Stop it manually.

      Note

      Stop the task during off-peak hours or when you are ready to switch to the destination instance.

      1. Wait until the task status changes to Incremental Data Migration and shows Undelayed. Then, stop writing data to the source database for a few minutes. The status of Incremental Data Migration may show a latency.

      2. Wait until the status of Incremental Data Migration changes back to Undelayed. Then, stop the migration task manually. The progress of schema migration and full data migration both show 100%. To pause the migration task, select the task and click Pause in the batch operations bar at the bottom.

12. Switch your business to the Redis instance.

Next steps

The database accounts used for data migration have read and write permissions. To ensure database security, change the passwords for both the self-managed Redis database and the Tair (Redis OSS-Compatible) instance after the migration is complete.