DocsICP FilingConsole
官方文档
首页

Migrate from a self-managed Redis to a Tair (Redis OSS-Compatible) instance

更新时间：
复制 MD 格式
产品详情
我的收藏

This topic describes how to use Data Transmission Service (DTS) to migrate data from a self-managed Redis database to a Tair (Redis OSS-Compatible) instance. DTS supports both full and incremental data migration. Combining these migration types allows you to smoothly migrate your self-managed Redis database to the cloud with minimal service interruption.

Prerequisites

  • You have created a source self-managed Redis instance and a destination Tair (Redis OSS-Compatible) instance. For more information about how to create a Tair (Redis OSS-Compatible) instance, see Step 1: Create an instance.

    Note

    • DTS currently supports only Tair (Redis OSS-Compatible) instances that use the direct connection mode.

    • For information about the supported versions of the source and destination databases, see Migration solutions overview.

  • The self-managed Redis can properly run the psync or sync commands.

  • The destination Tair (Redis OSS-Compatible) instance must have more available storage space than the amount used by the source self-managed Redis database.

Usage notes

Type

Description

Source database limits

  • Bandwidth requirement: The server hosting the source database must have sufficient outbound bandwidth. Otherwise, data migration speed will be affected.

  • Source database operation limits: If you perform only full data migration (during task configuration, set Migration Types to Full Data Migration only), do not write new data to the source instance. Otherwise, data inconsistency between source and destination will occur. To maintain real-time data consistency, select both full and incremental data migration.

  • Limits when migrating from a Basic Edition Redis source to a cluster architecture destination: Because Redis clusters allow only single-slot operations per command, multi-key operations that span multiple slots on the source will trigger the error:

    CROSSSLOT Keys in request don't hash to the same slot

    To avoid link interruption during DTS migration, perform only single-key operations.

  • If some keys in the source database use expiration (expire) policies, expired keys might not be immediately deleted. Therefore, the number of keys visible in the destination (for example, via the info command) may be fewer than in the source.

    Note

    The number of keys without expiration policies or unexpired keys is consistent between source and destination.

  • To ensure migration quality, DTS inserts a key with the prefix DTS_REDIS_TIMESTAMP_HEARTBEAT into the source database to record update timestamps. If the source uses a cluster architecture, DTS inserts this key into every shard. This key is filtered out during migration and expires after the task ends.

  • If the source database is a read-only instance or the DTS account lacks write (SETEX) permission, reported latency may be inaccurate.

  • To ensure migration stability based on Redis replication backlog mechanisms, increase the repl-backlog-size parameter in the redis.conf configuration file.

  • To migrate incremental data, the source account used for the task must have PSYNC and SYNC permissions.

Other limits

  • DTS consumes resources on both source and destination databases during full data migration, which may increase server load. If your database handles heavy workloads or runs on low-specification servers, this added pressure could even make the database service unavailable. Carefully evaluate your system before migration and run tasks during off-peak hours.

  • If the destination database runs out of memory and triggers data eviction, the default data eviction policy (maxmemory-policy) for Tair (Redis OSS-compatible) is volatile-lru. This can cause data inconsistency between source and destination but will not affect normal task execution.

    To prevent this, set the destination database's eviction policy to noeviction. When memory is insufficient, write operations will fail and the task will stop, but no data will be evicted from the destination.

    Note

    For details about data eviction policies, see Introduction to Redis data eviction policies.

  • For Lua scripts invoked via EVAL or EVALSHA, DTS cannot confirm successful execution during incremental migration because the destination does not explicitly return script execution results.

  • For List data structures, DTS does not execute Flush on existing data at the destination when using psync or sync to transfer data. This may result in duplicate data.

  • If scaling (adding or removing shards) or specification changes (such as increasing memory) occur on the self-managed Redis during migration, reconfigure the task. To ensure data consistency, clear all data already migrated to the destination Redis before reconfiguration.

  • If scaling (adding or removing shards), instance specification changes (such as increasing memory, changing specifications or configurations, or zone migration), or endpoint and port modifications occur on the source or destination Redis during migration, DTS cannot obtain continuous log data or correct connection information. This will interrupt the migration task. If this occurs, clear all data already synchronized to the destination Redis before reconfiguring the task to ensure data consistency.

  • After a failover event (including master/standby switchover) on the destination Redis, data might be written only to memory without being replicated to the secondary database, causing data loss.

  • For failed migration tasks, DTS triggers automatic recovery. Before switching your business to the destination instance, stop or release the task to prevent source data from overwriting destination data if the task auto-recovers.

  • If the destination instance uses a cluster architecture and any shard reaches its memory limit, or if the destination instance runs out of storage space, the DTS task fails due to out-of-memory (OOM) errors.

  • If transparent data encryption (TDE) is enabled on the destination instance, DTS does not support data migration.

  • If any of the following events occur during migration, full data may be re-migrated to the destination, causing data inconsistency:

    • Transient disconnections on the source or destination Redis cause breakpoint resume failure.

    • Master/standby switchover or failover occurs on the source or destination Redis.

    • The Redis endpoint changes on the source or destination.

  • If a Tair (Redis OSS-Compatible) instance has Transport Layer Security (TLS) encryption enabled, you must use an SSL-encrypted to connect to DTS (TLSv1.3 is not supported). Currently, you cannot connect a Tair (Redis OSS-Compatible) instance that has SSL enabled to DTS by using the Alibaba Cloud Instance method.

  • If a migration instance includes both full and incremental tasks, restarting the instance may cause DTS to re-execute both tasks.

  • Restarting a migration instance may cause some commands to execute repeatedly. This is especially true for non-idempotent commands (such as INCRBY or LPUSH) or when full migration restarts, potentially causing data inconsistency between source and destination.

  • If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust its parameters.

    Note

    Only DTS task parameters are modified—not database parameters. Parameters that may be adjusted include those listed in Modify instance parameters.

Billing

Migration type

Instance configuration fee

Internet traffic fee

Full data migration

Free of charge.

When the Access Method parameter of the destination database is set to Public IP Address, you are charged for Internet traffic. For more information, see Billing overview.

Incremental data migration

Charged. For more information, see Billing overview.

Migration types

  • Full data migration

    DTS migrates all existing data of the selected objects from the self-managed Redis database to the destination Tair (Redis OSS-Compatible) instance.

    Note

    If you perform only a full data migration, do not write new data to the self-managed Redis database during the migration to ensure data consistency.

  • Incremental data migration

    In addition to the full data migration, DTS migrates incremental data updates from the self-managed Redis database to the destination Tair (Redis OSS-Compatible) instance. Incremental data migration allows for a smooth migration to the cloud without service interruptions.

Commands supported for incremental data migration

  • APPEND

  • BITOP, BLPOP, BRPOP, BRPOPLPUSH

  • DECR, DECRBY, DEL

  • EVAL, EVALSHA, EXEC, EXPIRE, EXPIREAT

  • FLUSHALL, FLUSHDB

  • GEOADD, GETSET

  • HDEL, HINCRBY, HINCRBYFLOAT, HMSET, HSET, HSETNX

  • INCR, INCRBY, INCRBYFLOAT

  • LINSERT, LPOP, LPUSH, LPUSHX, LREM, LSET, LTRIM

  • MOVE, MSET, MSETNX, MULTI

  • PERSIST, PEXPIRE, PEXPIREAT, PFADD, PFMERGE, PSETEX, PUBLISH

  • RENAME, RENAMENX, RESTORE, RPOP, RPOPLPUSH, RPUSH, RPUSHX

  • SADD, SDIFFSTORE, SELECT, SET, SETBIT, SETEX, SETNX, SETRANGE, SINTERSTORE, SMOVE, SPOP, SREM, SUNIONSTORE

  • ZADD, ZINCRBY, ZINTERSTORE, ZREM, ZREMRANGEBYLEX, ZUNIONSTORE, ZREMRANGEBYRANK, ZREMRANGEBYSCORE

  • XADD, XCLAIM, XDEL, XAUTOCLAIM, XGROUP CREATECONSUMER, XTRIM

Preparations for incremental data migration

To ensure that the incremental data migration task runs as expected, remove the limit on the replication output buffer. This topic uses a server that runs Linux as an example.

Note

If you are performing only a full data migration, you can skip this step.

  1. Use the redis-cli tool to connect to the self-managed Redis database.

    Note

    The redis-cli tool is available after you install Redis. For more information, see the official Redis community website.

    redis-cli -h <host> -p <port> -a <password>
    Note

    • <host>: The endpoint of the self-managed Redis database. You can use 127.0.0.1.

    • <port>: The service port of the self-managed Redis database. The default port is 6379.

    • <password>: The password of the self-managed Redis database.

    Example:

    redis-cli -h 127.0.0.1 -p 6379 -a Test123456

  2. Run the following command to remove the limit on the replication output buffer: 

    config set client-output-buffer-limit 'slave 0 0 0'

Procedure

  1. Navigate to the migration task list page for the destination region using one of the following methods.

    From the DTS console

    1. Log on to the Data Transmission Service (DTS) console.

    2. In the navigation pane on the left, click Data Migration.

    3. In the upper-left corner of the page, select the region where the migration instance is located.

    From the DMS console

    Note

    The actual operations may vary based on the mode and layout of the DMS console. For more information, see Simple mode console and Customize the layout and style of the DMS console.

    1. Log on to the Data Management (DMS) console.

    2. In the top menu bar, choose Data + AI > Data Transmission (DTS) > Data Migration.

    3. To the right of Data Migration Tasks, select the region where the migration instance is located.

  2. Click Create Task to navigate to the task configuration page.

  3. Configure the source and destination databases.

    Category

    Parameter

    Description

    N/A

    Task Name

    DTS automatically generates a task name. We recommend that you specify a descriptive name for easy identification. The name does not need to be unique.

    Source Database

    Select Existing Connection

    • To use a database instance that has been added to the system (created or saved), select the desired database instance from the drop-down list. The database information below will be automatically configured.

      Note

      In the DMS console, this parameter is named Select a DMS database instance..

    • If you have not registered the database instance with the system, or do not need to use a registered instance, manually configure the database information below.

    Database Type

    Select Tair/Redis.

    Access Method

    Select an access method based on your source database's deployment location. This example uses Self-managed Database on ECS.

    Note

    If the source instance is a self-managed database, you must also complete the required preparations.

    Instance Region

    Select the region where the ECS instance that hosts the source Redis database resides.

    Replicate Data Across Alibaba Cloud Accounts

    In this example, data is migrated within the same Alibaba Cloud account. Select No.

    ECS Instance ID

    Select the ID of the ECS instance that hosts the source Redis database.

    Note

    If the source Redis database is in a cluster architecture, select the ECS instance ID where any master node resides. You must also manually add the DTS server CIDR block for the region to the security group rules of all other cluster instances. For more information, see Create a security group, Associate a security group with an instance (primary ENI), and Add the CIDR blocks for DTS servers to an IP address whitelist.

    Instance Mode

    Select Standalone or Cluster based on the architecture of the source Redis database.

    Note

    If the Access Method for the self-managed Redis is Public IP Address, the Cluster mode is not supported for Instance Mode.

    Port

    Enter the service port of the source Redis database. The default value is 6379.

    Note

    If the source Redis database is in a cluster architecture, enter the service port of any master node.

    Authentication Method

    Select an authentication method based on your requirements. In this example, Password Login is selected.

    Note

    • Only Redis 6.0 and later support Account + Password Login.

    • If you select Secret-free login, make sure that password-free access is enabled for the Redis database. For information about how to enable password-free access for a Tair (Redis OSS-Compatible) instance, see Enable password-free access over a VPC.

    Database Password

    Enter the password to connect to the source Redis database.

    Note

    • This parameter is optional. You can leave it empty if no password is set.

    • The password must be in the <user>:<password> format. For example, if the custom username for the Redis instance is admin and the password is Rp829dlwa, enter admin:Rp829dlwa.

    Encryption

    Select Non-encrypted or SSL-encrypted based on your database instance requirements.

    Note

    If you are using a self-managed Redis instance (the Access Method is not Alibaba Cloud Instance) and you select SSL-encrypted, you must also upload a CA Certificate and enter the CA Key.

    Destination Database

    Select Existing Connection

    • To use a database instance that has been added to the system (created or saved), select the desired database instance from the drop-down list. The database information below will be automatically configured.

      Note

      In the DMS console, this parameter is named Select a DMS database instance..

    • If you have not registered the database instance with the system, or do not need to use a registered instance, manually configure the database information below.

    Database Type

    Select Tair/Redis.

    Access Method

    Select Alibaba Cloud Instance.

    Instance Region

    Select the region where the destination Tair (Redis OSS-Compatible) instance resides.

    Replicate Data Across Alibaba Cloud Accounts

    In this example, data is migrated within the same Alibaba Cloud account. Select No.

    Instance ID

    Select the ID of the destination Tair (Redis OSS-Compatible) instance.

    Authentication Method

    Select an authentication method based on your requirements. In this example, Password Login is selected.

    Note

    • Only Redis 6.0 and later support Account + Password Login.

    • If you select Secret-free login, make sure that password-free access is enabled for the Redis database. For information about how to enable password-free access for a Tair (Redis OSS-Compatible) instance, see Enable password-free access over a VPC.

    Database Password

    Enter the password to connect to the destination Tair (Redis OSS-Compatible) instance.

    Note

    The password must be in the <user>:<password> format. For example, if the custom username for the Tair instance is admin and the password is Rp829dlwa, enter admin:Rp829dlwa.

    Encryption

    Select Non-encrypted or SSL-encrypted based on your database instance requirements.

    Note

    If you are using a self-managed Redis instance (the Access Method is not Alibaba Cloud Instance) and you select SSL-encrypted, you must also upload a CA Certificate and enter the CA Key.

  4. After you complete the configuration, click Test Connectivity and Proceed at the bottom of the page.

    Note

    • Ensure that the IP address segment of the DTS service is automatically or manually added to the security settings of the source and destination databases to allow access from DTS servers. For more information, see Add DTS server IP addresses to a whitelist.

    • If the source or destination database is a self-managed database (the Access Method is not Alibaba Cloud Instance), you must also click Test Connectivity in the CIDR Blocks of DTS Servers dialog box that appears.

  5. Configure the task objects.

    1. On the Configure Objects page, configure the objects that you want to migrate.

      Parameter

      Description

      Migration Types

      Select Full Data Migration + Incremental Data Migration.

      Note

      If the account does not have SYNC or PSYNC permissions on the source database, select Full Data Migration.

      Processing Mode of Conflicting Tables

      • Precheck and Report Errors: Checks whether the destination is empty. If the destination to be migrated to is empty, the precheck is passed. If not, an error is reported during the precheck, and the data migration job does not start.

      • Ignore Errors and Proceed: Skips the Check the existence of objects in the destination database. check item.

        Warning

        If you select Ignore Errors and Proceed and a key in the destination has the same name as a key in the source during migration, the data from the source overwrites the data in the destination. This may cause data loss in the destination. Proceed with caution.

      Source Objects

      In the Source Objects box, click the objects to migrate, and then click Right arrow to move them to the Selected Objects box.

      Note

      You can select objects at the database level. If you need to migrate data for specific keys within a database, you can use the data filtering feature in the Selected Objects pane.

      Selected Objects

      To specify a destination database (DB 0 to DB 255) or filter data by prefix, use the mapping and filtering functions. In the Selected Objects box, right-click the database to migrate, and then configure the settings in the Edit Schema dialog box. For more information, see Map object names and Set filter conditions.

    2. Click Next: Advanced Settings.

      Parameter

      Description

      Dedicated Cluster for Task Scheduling

      By default, DTS schedules tasks on a shared cluster. You do not need to select one. If you want more stable tasks, you can purchase a dedicated cluster to run DTS migration tasks.

      Retry Time for Failed Connections

      After the migration task starts, if the connection to the source or destination database fails, DTS reports an error and immediately begins to retry the connection. The default retry duration is 720 minutes. You can customize the retry time to a value from 10 to 1440 minutes. We recommend that you set the duration to more than 30 minutes. If DTS reconnects to the source and destination databases within the specified duration, the migration task automatically resumes. Otherwise, the task fails.

      Note

      • For multiple DTS instances that share the same source or destination, the network retry time is determined by the setting of the last created task.

      • Because you are charged for the task during the connection retry period, we recommend that you customize the retry time based on your business needs, or release the DTS instance as soon as possible after the source and destination database instances are released.

      Retry Time for Other Issues

      After the migration task starts, if a non-connectivity issue, such as a DDL or DML execution exception, occurs in the source or destination database, DTS reports an error and immediately begins to retry the operation. The default retry duration is 10 minutes. You can customize the retry time to a value from 1 to 1440 minutes. We recommend that you set the duration to more than 10 minutes. If the related operations succeed within the specified retry duration, the migration task automatically resumes. Otherwise, the task fails.

      Important

      The value of Retry Time for Other Issues must be less than the value of Retry Time for Failed Connections.

      Enable Throttling for Full Data Migration

      During full migration, DTS consumes read and write resources on the source and destination databases, which may increase the database load. If required, you can enable throttling for the full migration task. You can set Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s) to reduce the load on the destination database.

      Note

      This parameter is available only if you select Full Data Migration for Migration Types.

      Enable Throttling for Incremental Data Migration

      If required, you can also choose to set speed limits for the incremental migration task. You can set RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s) to reduce the load on the destination database.

      Note

      This parameter is available only if you select Incremental Data Migration for Migration Types.

      Extend Expiration Time of Destination Database Key

      Set an additional expiration time for keys when they are migrated from the source database to the destination database. To ensure data consistency, we recommend setting an extended key expiration time if commands such as the following are used.

      Note

      In scenarios that involve distributed locks, this may prevent locks from being released in a timely manner.

      expire key seconds
pexpire key milliseconds
expireat key timestamp
pexpireat key timestampMs

      Use Slave Node

      If the Instance Mode of the source self-managed Redis is Cluster, you can choose to read data from master or replica nodes. The default is No, which means data is read from master nodes.

      Environment Tag

      You can select an environment tag to identify the instance based on your requirements. In this example, no selection is needed.

      Configure ETL

      Choose whether to enable the extract, transform, and load (ETL) feature. For more information, see What is ETL? Valid values:

      Monitoring and Alerting

      Select whether to set alerts and receive alert notifications based on your business needs.

      • No: Does not set an alert.

      • Yes: Configure alerts by setting an alert threshold and an alert contact. If a migration fails or the latency exceeds the threshold, the system sends an alert notification.

    3. Click Data Verification to configure a data verification task.

      To use the data verification feature, see Configure data verification.

  6. Save the task and run a precheck.

    • To view the parameters for configuring this instance when you call the API operation, move the pointer over the Next: Save Task Settings and Precheck button and click Preview OpenAPI parameters in the bubble that appears.

    • If you do not need to view or have finished viewing the API parameters, click Next: Save Task Settings and Precheck at the bottom of the page.

    Note

    • Before the migration task starts, DTS performs a precheck. The task starts only after it passes the precheck.

    • If the precheck fails, click View Details next to the failed check item, fix the issue based on the prompt, and then run the precheck again.

    • If a warning is reported during the precheck:

      • For check items that cannot be ignored, click View Details next to the failed item, fix the issue based on the prompt, and then run the precheck again.

      • For check items that can be ignored, you can click Confirm Alert Details, Ignore, OK, and Precheck Again to skip the alert item and run the precheck again. If you choose to ignore a warning, it may cause issues such as data inconsistency and pose risks to your business.

  7. Purchase the instance.

    1. When the Success Rate reaches 100%, click Next: Purchase Instance.

    2. On the Purchase page, select the link specification for the data migration instance. For more information, see the following table.

      Category

      Parameter

      Description

      New Instance Class

      Resource Group Settings

      Select the resource group to which the instance belongs. The default value is default resource group. For more information, see What is Resource Management?

      Instance Class

      DTS provides migration specifications with different performance levels. The link specification affects the migration speed. You can select a specification based on your business scenario. For more information, see Data migration link specifications.

    3. After the configuration is complete, read and select Data Transmission Service (Pay-as-you-go) Service Terms.

    4. Click Buy and Start, and then click OK in the OK dialog box.

      You can view the specific progress on the data migration page.

      Note

      If your DTS instance is configured for both full and incremental migration (Migration Types includes both Full Data Migration and Incremental Data Migration), the task is listed as Incremental Data Migration on the migration task list page.

Previous：Cross-account migrationNext：Migrate data from ApsaraDB for Tair (compatible with Redis) to Message Queue for Apache Kafka