DocsICP FilingConsole
官方文档
首页

Sync Tair (Redis-compatible) to Message Queue for Kafka

更新时间：
复制 MD 格式
产品详情
我的收藏

This topic shows you how to use Data Transmission Service (DTS) to synchronize data from a source ApsaraDB for Tair (Redis-compatible) instance to a destination Message Queue for Apache Kafka instance.

Important

This feature is in beta and available only to select users.

Prerequisites

  • You have created a destination Message Queue for Apache Kafka instance. For instructions, see Procedure.

    Note

    For information about the supported versions of the source and destination databases, see Migration scenarios.

  • You have created a topic in the destination Message Queue for Apache Kafka instance to receive data. For instructions, see Step 1: Create a topic.

Precautions

Type

Description

Source database limitations

  • To ensure synchronization quality, Data Transmission Service (DTS) inserts a key with the prefix DTS_REDIS_TIMESTAMP_HEARTBEAT into the source database to record update timestamps. If the source database uses a cluster architecture, DTS inserts this key into each shard. This key is filtered out during synchronization and expires after the task is complete.

  • If the source database is a read-only instance or the account used by DTS does not have write (SETEX) permissions, the reported latency may be inaccurate.

  • To ensure a stable synchronization link that relies on the Redis replication backlog mechanism, we recommend that you increase the value of the repl-backlog-size parameter in the redis.conf configuration file.

  • If some keys in the source database use an expiration policy, the number of keys in the target database may be lower than that in the source database. This is because expired keys may not be immediately deleted from the source.

    Note

    The count of non-expiring and unexpired keys will be the same in both the source and target databases.

  • If a Tair (Redis OSS-Compatible) instance has Transport Layer Security (TLS) encryption enabled, you must connect to DTS using an SSL-encrypted. TLSv1.3 is not supported. You cannot connect to an SSL-enabled Tair (Redis OSS-Compatible) instance if you set the Alibaba Cloud Instance to Alibaba Cloud Instance.

  • If Transparent Data Encryption (TDE) is enabled on the source instance, DTS does not currently support data synchronization.

  • If the source is a Tair (Enterprise Edition) instance whose storage medium is persistent memory, ensure that the appendonly parameter is set to yes. For more information, see Procedure.

  • If the source Redis instance uses Redis extension modules, such as RedisJSON, RediSearch, RedisBloom, RedisTimeSeries, RedisGraph, RedisAI, RedisGears, RedisCell, RediSQL, redis-tdigest, or RedisCompat, the task will fail because DTS cannot parse their RDB storage formats.

  • If the source is a self-managed Redis instance:

    • If the endpoint of the self-managed Redis instance changes during synchronization, you must reconfigure the task.

    • We recommend setting the repl-timeout parameter, which specifies the replication timeout between the replica and master, to 600 seconds by using the config set repl-timeout 600 command. If the source database contains a large amount of data, you can increase the value of the repl-timeout parameter as needed.

  • Certain conditions can trigger a full data re-synchronization to the target, which may lead to data inconsistency. These conditions include:

    • A transient connection interruption on the source Redis instance causes resumable transmission to fail.

    • A failover occurs on the source Redis instance.

    • The endpoint of the source Redis instance changes.

    • The synchronization objects of the DTS instance are modified.

  • During synchronization, operations such as scaling (adding or removing shards), instance configuration changes (such as increasing memory, changing specifications, and availability zone migration), or changing the connection address and port on either the source or destination Redis instance will interrupt the synchronization task because DTS can no longer access continuous log data or correct connection information. If this happens, you must reconfigure the task. To ensure data consistency, we recommend that you clear all data from the destination Redis instance before you reconfigure the task.

  • Due to feature limitations, DTS synchronizes data only from the selected child instance of the global multi-active feature for Tair (Enterprise Edition), not from other child instances in the group.

    Note

    If you need to synchronize data from other child instances in the global multi-active group, you can join the DTS customer communication group on DingTalk (Group ID: 116655009709) for assistance. (Download DingTalk)

Other limitations

  • You must restart the DTS instance if the target Kafka instance is scaled out or in (for example, by adding or removing broker nodes) while the task is running.

  • Ensure that DTS can connect to both the source and target instances. For example, ensure that access from DTS is not restricted by security settings of the database instance, the bind parameter in the redis.conf file for a self-managed Redis instance, or the listeners and advertised.listeners parameters in the server.properties file for a self-managed Kafka instance.

  • During a full task, DTS increases the load on both the source database and target instance by consuming their read and write resources. Before you run the DTS instance, evaluate the performance of both the source and target instances. We recommend running the instance during off-peak hours, for example, when the CPU utilization of both is below 30%.

  • Writing data to the target instance from any source other than DTS while the task is running can cause data inconsistency or task failure.

  • DTS may attempt to recover failed tasks for up to seven days. Therefore, before switching your services to the target instance, you must end or release the DTS task. This prevents the task from being automatically recovered and overwriting data in the target instance.

  • If a synchronization instance includes both a full synchronization task and an incremental synchronization task, restarting the instance may cause DTS to re-run both tasks.

  • Restarting a synchronization instance may cause some commands to be re-executed. This can lead to data inconsistency between the source and destination databases, especially if non-idempotent commands, such as INCRBY and LPUSH, are used or if a full synchronization task is re-triggered.

  • If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust its parameters.

    Note

    Only DTS task parameters are modified—not database parameters. Parameters that may be adjusted include those listed in Modify instance parameters.

Billing

Synchronization type

Configuration fee

Full data synchronization

Free.

Incremental data synchronization

Charged. For details, see billing overview.

Supported commands

  • APPEND

  • BITOP, BLPOP, BRPOP, and BRPOPLPUSH

  • DECR, DECRBY, and DEL

  • EVAL, EVALSHA, EXEC, EXPIRE, and EXPIREAT

  • GEOADD and GETSET

  • HDEL, HINCRBY, HINCRBYFLOAT, HMSET, HSET, and HSETNX

  • INCR, INCRBY, and INCRBYFLOAT

  • LINSERT, LPOP, LPUSH, LPUSHX, LREM, LSET, and LTRIM

  • MOVE, MSET, MSETNX, and MULTI

  • PERSIST, PEXPIRE, PEXPIREAT, PFADD, PFMERGE, and PSETEX

  • RENAME, RENAMENX, RESTORE, RPOP, RPOPLPUSH, RPUSH, and RPUSHX

  • SADD, SDIFFSTORE, SELECT, SET, SETBIT, SETEX, SETNX, SETRANGE, SINTERSTORE, SMOVE, SPOP, SREM, and SUNIONSTORE

  • ZADD, ZINCRBY, ZINTERSTORE, ZREM, ZREMRANGEBYLEX, ZUNIONSTORE, ZREMRANGEBYRANK, and ZREMRANGEBYSCORE

  • SWAPDB and UNLINK (supported only when the source Redis instance is version 4.0)

  • XADD, XCLAIM, XDEL, XAUTOCLAIM, XGROUP CREATECONSUMER, and XTRIM

Note

  • The PUBLISH command is not supported.

  • DTS cannot guarantee the successful execution of a Lua script called with EVAL or EVALSHA during incremental data synchronization because the destination does not explicitly return an execution result.

  • When retransmitting list data using SYNC or PSYNC, DTS does not clear existing data in the destination, which may cause duplicate data.

Database account permissions

Database

Required permissions

Creation and authorization

Source Tair (Redis-compatible) instance

Read permission

Create and manage accounts

Procedure

  1. Go to the data synchronization task list page in the destination region. You can do this in one of two ways.

    DTS console

    1. Log on to the DTS console.

    2. In the navigation pane on the left, click Data Synchronization.

    3. In the upper-left corner of the page, select the region where the synchronization instance is located.

    DMS console

    Note

    The actual steps may vary depending on the mode and layout of the DMS console. For more information, see Simple mode console and Customize the layout and style of the DMS console.

    1. Log on to the DMS console.

    2. In the top menu bar, choose Data + AI > DTS (DTS) > Data Synchronization.

    3. To the right of Data Synchronization Tasks, select the region of the synchronization instance.

  2. Click Create Task to open the task configuration page.

  3. Configure the source and destination databases.

    Section

    Parameter

    Description

    None

    Task Name

    DTS automatically generates a task name. We recommend that you specify a descriptive name for easy identification. The name does not need to be unique.

    Source Database

    Select Existing Connection

    • Select the registered database instance with DTS from the drop-down list. The database information below is automatically configured.

      Note

      In the DMS console, this configuration item is Select a DMS database instance.

    • If you have not registered the database instance or do not need to use a registered instance, manually configure the database information below.

    Database Type

    Select Tair/Redis.

    Access Method

    Select Alibaba Cloud Instance.

    Instance Region

    Select the region where the source Tair (Redis OSS-Compatible) instance resides.

    Replicate Data Across Alibaba Cloud Accounts

    For this example, select No, as the database instance belongs to the current Alibaba Cloud account.

    Instance ID

    Select the ID of the source Tair (Redis OSS-Compatible) instance.

    Authentication Method

    Select an authentication method based on your requirements. In this example, Password Login is selected.

    Note

    • Only Redis 6.0 and later support Account + Password Login.

    • If you select Secret-free login, make sure that password-free access is enabled for the Redis database. For information about how to enable password-free access for a Tair (Redis OSS-Compatible) instance, see Enable password-free access over a VPC.

    Database Password

    Enter the password for the account used to connect to the source Tair (Redis OSS-Compatible) instance. This account must have read permissions.

    • Default account: Enter the password.

    • For a newly created account, the password format is <username>:<password>. For example, if the custom username for a Redis instance is admin and the password is Rp829dlwa, the database password is admin:Rp829dlwa.

    Encryption

    Select Non-encrypted or SSL-encrypted based on your database instance requirements.

    Note

    If you are using a self-managed Redis instance (the Access Method is not Alibaba Cloud Instance) and you select SSL-encrypted, you must also upload a CA Certificate and enter the CA Certificate Password.

    Destination Database

    Select Existing Connection

    • Select the registered database instance with DTS from the drop-down list. The database information below is automatically configured.

      Note

      In the DMS console, this configuration item is Select a DMS database instance.

    • If you have not registered the database instance or do not need to use a registered instance, manually configure the database information below.

    Database Type

    Select Kafka.

    Access Method

    Select Alibaba Cloud Instance.

    Instance Region

    Select the region where the destination Kafka instance resides.

    Kafka Instance ID

    Select the ID of the destination Kafka instance.

    Encryption

    Select Non-encrypted or SCRAM-SHA-256 based on your business and security requirements.

    Topic

    From the drop-down list, select the topic to receive data.

    Use Kafka Schema Registry

    Kafka Schema Registry is a metadata serving layer that provides a RESTful interface to store and retrieve your Avro schemas.

    • No: Do not use Kafka Schema Registry.

    • Yes: Use Kafka Schema Registry. You must enter the URL or IP address that is registered in Kafka Schema Registry for your Avro schemas.

  4. After completing the configuration, click Test Connectivity and Proceed at the bottom of the page.

    Note

    • Ensure that you add the CIDR blocks of the DTS servers (either automatically or manually) to the security settings of both the source and destination databases to allow access. For more information, see Add the IP address whitelist of DTS servers.

    • If the source or destination is a self-managed database (i.e., the Access Method is not Alibaba Cloud Instance), you must also click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

  5. Configure the task objects.

    1. On the Configure Objects page, specify the objects to synchronize.

      Parameter

      Description

      Synchronization Types

      Incremental Data Synchronization is selected by default. We recommend also selecting Full Data Synchronization.

      Processing Mode of Conflicting Tables

      Retain the default settings.

      Data Format in Kafka

      Select the storage format for data to be synchronized to the Kafka instance.

      Kafka Data Compression Format

      Select a compression format for Kafka messages based on your business requirements.

      • LZ4 (Default): offers a low compression ratio and high compression speed.

      • GZIP: offers a high compression ratio and low compression speed.

        Note

        GZIP compression consumes significant CPU resources.

      • Snappy: offers a medium compression ratio and medium compression speed.

      Policy for Shipping Data to Kafka Partitions

      Select a synchronization policy for partitions based on your business requirements. For more information, see Kafka partition synchronization policies.

      Message acknowledgement mechanism

      Select a message acknowledgment mechanism based on your business requirements.

      Topic That Stores DDL Information
      From the drop-down list, select the topic that is used to store DDL information. Note: If you do not select a topic, DDL information is stored by default in the topic that receives the data.

      Source Objects

      In the Source Objects box, click the objects, and then click 向右 to move them to the Selected Objects box.

      Note

      You can select objects only at the database level. To synchronize data for specific keys within a database, use the data filtering feature in the Selected Objects pane.

      Selected Objects

      In this example, no additional configuration is required.

      You can use the mapping feature to configure the topic name, the number of partitions in the topic, and additional columns for the source database (DB 0 to DB 255) in the destination Kafka instance. For more information, see Configure mapping information.

    2. Click Next: Advanced Settings.

      Parameter

      Description

      Dedicated Cluster for Task Scheduling

      By default, DTS uses a shared cluster for tasks, so you do not need to make a selection. For greater task stability, you can purchase a dedicated cluster to run the DTS synchronization task. For more information, see What is a DTS dedicated cluster?.

      Retry Time for Failed Connections

      If the connection to the source or destination database fails after the synchronization task starts, DTS reports an error and immediately begins to retry the connection. The default retry duration is 720 minutes. You can customize the retry time to a value from 10 to 1,440 minutes. We recommend a duration of 30 minutes or more. If the connection is restored within this period, the task resumes automatically. Otherwise, the task fails.

      Note

      • If multiple DTS instances (e.g., Instance A and B) share a source or destination, DTS uses the shortest configured retry duration (e.g., 30 minutes for A, 60 for B, so 30 minutes is used) for all instances.

      • DTS charges for task runtime during connection retries. Set a custom duration based on your business needs, or release the DTS instance promptly after you release the source/destination instances.

      Retry Time for Other Issues

      If a non-connection issue (e.g., a DDL or DML execution error) occurs, DTS reports an error and immediately retries the operation. The default retry duration is 10 minutes. You can also customize the retry time to a value from 1 to 1,440 minutes. We recommend a duration of 10 minutes or more. If the related operations succeed within the set retry time, the synchronization task automatically resumes. Otherwise, the task fails.

      Important

      The value of Retry Time for Other Issues must be less than that of Retry Time for Failed Connections.

      Enable Throttling for Full Data Synchronization

      During full data synchronization, DTS consumes read and write resources from the source and destination databases, which can increase their load. To mitigate pressure on the destination database, you can limit the migration rate by setting Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s).

      Note

      This parameter is available only when Synchronization Types is selected for Full Data Synchronization.

      Enable Throttling for Incremental Data Synchronization

      You can also limit the incremental synchronization rate to reduce pressure on the destination database by setting RPS of Incremental Data Synchronization and Data synchronization speed for incremental synchronization (MB/s).

      Use Slave Node

      In this example, no configuration is required.

      If the source database is a self-managed Redis instance and Instance Mode is set to Cluster, you can choose to read data from the master or slave nodes. The default value is No, which means DTS reads data from the master nodes.

      Environment Tag

      You can select an environment tag to identify the instance based on your requirements. In this example, no environment tag is selected.

      Configure ETL

      Choose whether to enable the extract, transform, and load (ETL) feature. For more information, see What is ETL? Valid values:

      Monitoring and Alerting

      Choose whether to set up alerts. If the synchronization fails or the latency exceeds the specified threshold, DTS sends a notification to the alert contacts.

  6. Save the task and perform a precheck.

    • To view the parameters for configuring this instance via an API operation, hover over the Next: Save Task Settings and Precheck button and click Preview OpenAPI parameters in the tooltip.

    • If you have finished viewing the API parameters, click Next: Save Task Settings and Precheck at the bottom of the page.

    Note

    • Before a synchronization task starts, DTS performs a precheck. You can start the task only if the precheck passes.

    • If the precheck fails, click View Details next to the failed item, fix the issue as prompted, and then rerun the precheck.

    • If the precheck generates warnings:

      • For non-ignorable warning, click View Details next to the item, fix the issue as prompted, and run the precheck again.

      • For ignorable warnings, you can bypass them by clicking Confirm Alert Details, then Ignore, and then OK. Finally, click Precheck Again to skip the warning and run the precheck again. Ignoring precheck warnings may lead to data inconsistencies and other business risks. Proceed with caution.

  7. Purchase the instance.

    1. When the Success Rate reaches 100%, click Next: Purchase Instance.

    2. On the Purchase page, select the billing method and link specifications for the data synchronization instance. For more information, see the following table.

      Category

      Parameter

      Description

      New Instance Class

      Billing Method

      • Subscription: You pay upfront for a specific duration. This is cost-effective for long-term, continuous tasks.

      • Pay-as-you-go: You are billed hourly for actual usage. This is ideal for short-term or test tasks, as you can release the instance at any time to save costs.

      Resource Group Settings

      The resource group to which the instance belongs. The default is default resource group. For more information, see What is Resource Management?.

      Instance Class

      DTS offers synchronization specifications at different performance levels that affect the synchronization rate. Select a specification based on your business requirements. For more information, see Data synchronization link specifications.

      Subscription Duration

      In subscription mode, select the duration and quantity of the instance. Monthly options range from 1 to 9 months. Yearly options include 1, 2, 3, or 5 years.

      Note

      This option appears only when the billing method is Subscription.

    3. Read and select the checkbox for Data Transmission Service (Pay-as-you-go) Service Terms.

    4. Click Buy and Start. In the OK dialog box that appears, click OK.

      You can view the task progress on the Data Synchronization page.

      Note

      If you select both Full Data Synchronization and Incremental Data Synchronization for Synchronization Types, the instance is displayed as an Incremental Data Synchronization task on the synchronization task list page.

Configure mapping information

  1. In the Selected Objects section, right-click the selected synchronization object.

  2. In the Edit Schema dialog box, configure the following parameters.

    Parameter

    Description

    Name of target Topic

    The topic in the destination Kafka instance that receives data. This defaults to the Topic you selected in the Destination Database section during the Configurations for Source and Destination Databases step.

    Important

    • The topic name must exist in the destination Kafka instance. Otherwise, data synchronization fails.

    • If you modify the Name of target Topic, data will be written to the specified topic.

    Number of Partitions

    The number of partitions for the target topic.

    Prefixes of Keys to Be Synchronized

    For more information, see Set filter conditions.

    Prefixes of Keys to Be Filtered Out

    Additional Columns

    For more information, see Add additional columns.

  3. Click OK.

FAQ

Previous：Sync self-managed Redis on ECS to TairNext：Synchronize data from a TiDB database