DocsICP FilingConsole
Documentation
Home Page Data Transmission Service User Guide Data Synchronization (New Console) Synchronize data from a Redis database Sync Tair (Redis-compatible) to Message Queue for Kafka

Sync Tair (Redis-compatible) to Message Queue for Kafka

Updated at:
Copy as MD
Product
My Favorites

Data Transmission Service (DTS) supports synchronizing data from a source ApsaraDB for Tair (Redis-compatible) instance to a destination Message Queue for Apache Kafka instance. This topic describes the synchronization procedure.

Important

This feature is in beta and available only to select users.

Prerequisites

  • You have created a destination Message Queue for Apache Kafka instance. For instructions, see Procedure.

    Note

    For information about the supported versions of the source and destination databases, see Migration scenarios.

  • You have created a topic in the destination Message Queue for Apache Kafka instance to receive data. For instructions, see Step 1: Create a topic.

Precautions

Type

Description

Source database limitations

  • DTS inserts a key prefixed with DTS_REDIS_TIMESTAMP_HEARTBEAT into the source database to record update timestamps. For clusters, this key is inserted into each shard. The key is filtered during synchronization and expires after the task completes.

  • If the source database is a read-only instance or the DTS account does not have write (SETEX) permissions, the reported latency may be inaccurate.

  • To ensure connection stability, increase the value of the repl-backlog-size parameter in the redis.conf file.

  • If some keys in the source database have an expiration policy, they may not be deleted immediately upon expiry. The key count in the destination may be lower than in the source.

    Note

    The number of keys that do not have an expiration policy or have not yet expired is the same in both the source and destination databases.

  • If a Tair (Redis OSS-Compatible) instance has Transport Layer Security (TLS) encryption enabled, you must connect to it by using the SSL-encrypted method in DTS. TLSv1.3 is not supported. You cannot connect to an SSL-enabled Tair (Redis OSS-Compatible) instance by setting the Alibaba Cloud Instance to Alibaba Cloud Instance.

  • DTS does not support synchronization if Transparent Data Encryption (TDE) is enabled on the source.

  • If the source instance is a Tair (Enterprise Edition) instance whose Storage Medium is Persistent Memory, ensure that the appendonly parameter is set to yes as described in Steps.

  • If the source Redis instance uses extended modules such as RedisJSON, RediSearch, RedisBloom, RedisTimeSeries, RedisGraph, RedisAI, RedisGears, RedisCell, RediSQL, redis-tdigest, or RedisCompat, DTS cannot parse their RDB storage formats and the task fails.

  • If the source is a self-managed Redis instance:

    • If the self-managed Redis endpoint changes during synchronization, you must reconfigure the task.

    • Set repl-timeout to 600 seconds by running config set repl-timeout 600. Increase the repl-timeout value if the source contains a large amount of data.

  • Full data may be re-synchronized to the destination and cause data inconsistency under the following conditions:

    • A transient connection interruption occurs on the source Redis instance, and resumable transmission fails.

    • A primary/standby switchover or a failover occurs on the source Redis.

    • The endpoint of the source Redis instance changes.

    • The synchronized objects of the DTS instance are modified.

  • Operations on the source or target Redis instance during synchronization, such as scaling, changing specifications, migrating availability zones, or changing the endpoint or port, can interrupt the task because DTS cannot obtain continuous log data or correct connection information. If this occurs, clear the synchronized data from the target before you reconfigure the task.

  • If the source is a child instance of a Tair (Enterprise Edition) Global Distributed Cache, DTS can synchronize data only from the selected child instance. Data from other child instances in the group is not synchronized.

    Note

    To synchronize data from other child instances in the Global Distributed Cache group, join the DTS customer communication group on DingTalk (Group ID: 116655009709) for assistance. You can download the DingTalk client from the DingTalk client download link.

Other limitations

  • If the destination Kafka instance is scaled in or out, for example, by adding or removing broker nodes, while the DTS task is running, you must restart the DTS task.

  • Ensure that DTS can connect to the source and destination. For example, verify that the security settings of the database instance, the bind parameter in the self-managed Redis configuration file redis.conf, and the listeners and advertised.listeners parameters in the self-managed Kafka configuration file server.properties do not restrict access from DTS.

  • During initial full data synchronization, DTS consumes read and write resources on both the source and destination, which may increase load. Evaluate performance beforehand and run the task during off-peak hours, for example, when CPU utilization is below 30%.

  • If data is written to the destination from any source other than DTS while the DTS task is running, data inconsistency may occur, and the DTS task may fail.

  • DTS attempts to recover failed tasks for up to seven days. To prevent automatic recovery from overwriting data after you switch your services, you must end or release the task before the switch.

  • Restarting a synchronization instance that includes both full and incremental tasks may cause DTS to rerun both.

  • Restarting a synchronization instance may cause commands to execute more than once, leading to data inconsistency if non-idempotent commands such as INCRBY and LPUSH exist, or if full data re-synchronization is triggered.

  • If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust its parameters.

    Note

    Only DTS task parameters are modified—not database parameters. Parameters that may be adjusted include those listed in Modify instance parameters.

Billing

Type

Fee

Full data synchronization

Free.

Incremental data synchronization

Charged. For details, see Billing overview.

Supported commands

  • APPEND

  • BITOP, BLPOP, BRPOP, and BRPOPLPUSH

  • DECR, DECRBY, and DEL

  • EVAL, EVALSHA, EXEC, EXPIRE, and EXPIREAT

  • GEOADD and GETSET

  • HDEL, HINCRBY, HINCRBYFLOAT, HMSET, HSET, and HSETNX

  • INCR, INCRBY, and INCRBYFLOAT

  • LINSERT, LPOP, LPUSH, LPUSHX, LREM, LSET, and LTRIM

  • MOVE, MSET, MSETNX, and MULTI

  • PERSIST, PEXPIRE, PEXPIREAT, PFADD, PFMERGE, and PSETEX

  • RENAME, RENAMENX, RESTORE, RPOP, RPOPLPUSH, RPUSH, and RPUSHX

  • SADD, SDIFFSTORE, SELECT, SET, SETBIT, SETEX, SETNX, SETRANGE, SINTERSTORE, SMOVE, SPOP, SREM, and SUNIONSTORE

  • ZADD, ZINCRBY, ZINTERSTORE, ZREM, ZREMRANGEBYLEX, ZUNIONSTORE, ZREMRANGEBYRANK, and ZREMRANGEBYSCORE

  • SWAPDB and UNLINK (supported only when the source Redis instance is version 4.0)

  • XADD, XCLAIM, XDEL, XAUTOCLAIM, XGROUP CREATECONSUMER, and XTRIM

Note

  • The PUBLISH command is not supported.

  • DTS cannot guarantee the successful execution of a Lua script called with EVAL or EVALSHA during incremental data synchronization because the destination does not explicitly return an execution result.

  • When retransmitting list data using SYNC or PSYNC, DTS does not clear existing data in the destination, which may cause duplicate data.

Database account permissions

Database

Required permissions

Creation and authorization

Source Tair (Redis-compatible) instance

Read permission

Create and manage accounts

Procedure

  1. Go to the data synchronization task list page in the destination region. You can do this in one of two ways.

    DTS console

    1. Log on to the DTS console.

    2. In the navigation pane on the left, click Data Synchronization.

    3. In the upper-left corner of the page, select the region where the synchronization instance is located.

    DMS console

    Note

    The actual steps may vary depending on the mode and layout of the DMS console. For more information, see Simple mode console and Customize DMS console layout and style.

    1. Log on to the DMS console.

    2. In the top menu bar, choose Data + AI > DTS (DTS) > Data Synchronization.

    3. To the right of Data Synchronization Tasks, select the region of the synchronization instance.

  2. Click Create Task to open the task configuration page.

  3. Configure the source and destination databases.

    Section

    Parameter

    Description

    None

    Task Name

    DTS automatically generates a task name. We recommend that you specify a descriptive name for easy identification. The name does not need to be unique.

    Source Database

    Select Existing Connection

    • Select the registered database instance with DTS from the drop-down list. The database information below is automatically configured.

      Note

      In the DMS console, this configuration item is Select a DMS database instance.

    • If you have not registered the database instance or do not need to use a registered instance, manually configure the database information below.

    Database Type

    Select Tair/Redis.

    Access Method

    Select Alibaba Cloud Instance.

    Instance Region

    Select the region where the source Tair (Redis OSS-Compatible) instance resides.

    Replicate Data Across Alibaba Cloud Accounts

    For this example, select No, as the database instance belongs to the current Alibaba Cloud account.

    Instance ID

    Select the ID of the source Tair (Redis OSS-Compatible) instance.

    Authentication Method

    Select an authentication method based on your requirements. In this example, Password Login is selected.

    Note

    • Only Redis 6.0 and later support Account + Password Login.

    • If you select Secret-free login, make sure that password-free access is enabled for the Redis database. For information about how to enable password-free access for a Tair (Redis OSS-Compatible) instance, see Enable password-free access over a VPC.

    Database Password

    Enter the password of the account that is used to connect to the source Tair (Redis OSS-Compatible) instance. The account must have read permissions.

    • Default account: Enter the password.

    • For a newly created account, the password format is <username>:<password>. For example, if the custom username for a Redis instance is admin and the password is Rp829dlwa, the database password is admin:Rp829dlwa.

    Encryption

    Select Non-encrypted or SSL-encrypted based on your database instance requirements.

    Note

    If you are using a self-managed Redis instance (the Access Method is not Alibaba Cloud Instance) and you select SSL-encrypted, you must also upload a CA Certificate and enter the CA Certificate Password.

    Destination Database

    Select Existing Connection

    • Select the registered database instance with DTS from the drop-down list. The database information below is automatically configured.

      Note

      In the DMS console, this configuration item is Select a DMS database instance.

    • If you have not registered the database instance or do not need to use a registered instance, manually configure the database information below.

    Database Type

    Select Kafka.

    Access Method

    Select Alibaba Cloud Instance.

    Instance Region

    Select the region where the destination Kafka instance resides.

    Kafka Instance ID

    Select the ID of the destination Kafka instance.

    Encryption

    Select Non-encrypted or SCRAM-SHA-256 based on your business and security requirements.

    Topic

    Select the topic that receives the synchronized data.

    Use Kafka Schema Registry

    Kafka Schema Registry is a metadata serving layer that provides a RESTful interface to store and retrieve your Avro schemas.

    • No: Do not use Kafka Schema Registry.

    • Yes: Use Kafka Schema Registry. You must enter the URL or IP address that is registered in Kafka Schema Registry for your Avro schemas.

  4. After completing the configuration, click Test Connectivity and Proceed at the bottom of the page.

    Note

    • Ensure that you add the CIDR blocks of the DTS servers (either automatically or manually) to the security settings of both the source and destination databases to allow access. For more information, see Add the IP address whitelist of DTS servers.

    • If the source or destination is a self-managed database (i.e., the Access Method is not Alibaba Cloud Instance), you must also click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

  5. Configure the task objects.

    1. On the Configure Objects page, specify the objects to synchronize.

      Parameter

      Description

      Synchronization Types

      Incremental Data Synchronization is selected by default. We recommend also selecting Full Data Synchronization.

      Processing Mode of Conflicting Tables

      Retain the default settings.

      Data Format in Kafka

      Select the data storage format that is used when data is synchronized to the Kafka instance based on your business requirements.

      Kafka Data Compression Format

      Select the compression format for Kafka message compression based on your business requirements.

      • LZ4 (default): The compression ratio is low, but the compression speed is high.

      • GZIP: The compression ratio is high, but the compression speed is low.

        Note

        CPU consumption is high.

      • Snappy: The compression ratio and compression speed are medium.

      Policy for Shipping Data to Kafka Partitions

      Select a synchronization policy for partitions based on your business requirements. For more information, see Kafka partition synchronization policies.

      Message acknowledgement mechanism

      Select a message acknowledgment mechanism based on your business requirements.

      Topic That Stores DDL Information
      Select the topic that stores DDL information. If you do not select a topic, DDL information is stored in the topic that receives the data by default.

      Source Objects

      In the Source Objects box, click the objects, and then click 向右 to move them to the Selected Objects box.

      Note

      You can select objects only at the database level. To synchronize data for specific keys within a database, use the data filtering feature in the Selected Objects pane.

      Selected Objects

      In this example, no additional configuration is required.

      Use the mapping feature to configure the topic name, partition count, and additional columns for the source database (DB 0 to DB 255) in the destination Kafka instance. For more information, see Configure mapping information.

    2. Click Next: Advanced Settings.

      Parameter

      Description

      Dedicated Cluster for Task Scheduling

      By default, DTS uses a shared cluster for tasks, so you do not need to make a selection. For greater task stability, you can purchase a dedicated cluster to run the DTS synchronization task. For more information, see What is a DTS dedicated cluster?.

      Retry Time for Failed Connections

      If the connection to the source or destination database fails after the synchronization task starts, DTS reports an error and immediately begins to retry the connection. The default retry duration is 720 minutes. You can customize the retry time to a value from 10 to 1,440 minutes. We recommend a duration of 30 minutes or more. If the connection is restored within this period, the task resumes automatically. Otherwise, the task fails.

      Note

      • If multiple DTS instances (e.g., Instance A and B) share a source or destination, DTS uses the shortest configured retry duration (e.g., 30 minutes for A, 60 for B, so 30 minutes is used) for all instances.

      • DTS charges for task runtime during connection retries. Set a custom duration based on your business needs, or release the DTS instance promptly after you release the source/destination instances.

      Retry Time for Other Issues

      If a non-connection issue (e.g., a DDL or DML execution error) occurs, DTS reports an error and immediately retries the operation. The default retry duration is 10 minutes. You can also customize the retry time to a value from 1 to 1,440 minutes. We recommend a duration of 10 minutes or more. If the related operations succeed within the set retry time, the synchronization task automatically resumes. Otherwise, the task fails.

      Important

      The value of Retry Time for Other Issues must be less than that of Retry Time for Failed Connections.

      Enable Throttling for Full Data Synchronization

      During full data synchronization, DTS consumes read and write resources from the source and destination databases, which can increase their load. To mitigate pressure on the destination database, you can limit the migration rate by setting Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s).

      Note

      This parameter is available only when Synchronization Types is selected for Full Data Synchronization.

      Enable Throttling for Incremental Data Synchronization

      You can also limit the incremental synchronization rate to reduce pressure on the destination database by setting RPS of Incremental Data Synchronization and Data synchronization speed for incremental synchronization (MB/s).

      Use Slave Node

      In this example, no configuration is required.

      If the source database is a self-managed Redis instance and Instance Mode is set to Cluster, you can specify whether to read data from the master or slave nodes. The default value is No, which means DTS reads data from the master nodes.

      Environment Tag

      Select an environment tag to identify the instance. In this example, no environment tag is selected.

      Configure ETL

      Choose whether to enable the extract, transform, and load (ETL) feature. For more information, see What is ETL? Valid values:

      Monitoring and Alerting

      Choose whether to set up alerts. If the synchronization fails or the latency exceeds the specified threshold, DTS sends a notification to the alert contacts.

  6. Save the task and perform a precheck.

    • To view the parameters for configuring this instance via an API operation, hover over the Next: Save Task Settings and Precheck button and click Preview OpenAPI parameters in the tooltip.

    • If you have finished viewing the API parameters, click Next: Save Task Settings and Precheck at the bottom of the page.

    Note

    • Before a synchronization task starts, DTS performs a precheck. You can start the task only if the precheck passes.

    • If the precheck fails, click View Details next to the failed item, fix the issue as prompted, and then rerun the precheck.

    • If the precheck generates warnings:

      • For non-ignorable warning, click View Details next to the item, fix the issue as prompted, and run the precheck again.

      • For ignorable warnings, you can bypass them by clicking Confirm Alert Details, then Ignore, and then OK. Finally, click Precheck Again to skip the warning and run the precheck again. Ignoring precheck warnings may lead to data inconsistencies and other business risks. Proceed with caution.

  7. Purchase the instance.

    1. When the Success Rate reaches 100%, click Next: Purchase Instance.

    2. On the Purchase page, select the billing method and link specifications for the data synchronization instance. For more information, see the following table.

      Category

      Parameter

      Description

      New Instance Class

      Billing Method

      • Subscription: You pay upfront for a specific duration. This is cost-effective for long-term, continuous tasks.

      • Pay-as-you-go: You are billed hourly for actual usage. This is ideal for short-term or test tasks, as you can release the instance at any time to save costs.

      Resource Group Settings

      The resource group to which the instance belongs. The default is default resource group. For more information, see What is resource management?.

      Instance Class

      DTS offers synchronization specifications at different performance levels that affect the synchronization rate. Select a specification based on your business requirements. For more information, see Data synchronization link specifications.

      Subscription Duration

      In subscription mode, select the duration and quantity of the instance. Monthly options range from 1 to 9 months. Yearly options include 1, 2, 3, or 5 years.

      Note

      This option appears only when the billing method is Subscription.

    3. Read and select the checkbox for Data Transmission Service (Pay-as-you-go) Service Terms.

    4. Click Buy and Start. In the OK dialog box that appears, click OK.

      You can view the task progress on the Data Synchronization page.

      Note

      If you select both Full Data Synchronization and Incremental Data Synchronization for Synchronization Types, the instance is displayed as an Incremental Data Synchronization task on the synchronization task list page.

Configure mapping information

  1. In the Selected Objects section, right-click the selected synchronization object.

  2. In the Edit Schema dialog box, configure the following parameters.

    Parameter

    Description

    Name of target Topic

    The topic in the destination Kafka instance that receives data. This defaults to the Topic you selected in the Destination Database section during the Configurations for Source and Destination Databases step.

    Important

    • The topic name must exist in the destination Kafka instance. Otherwise, data synchronization fails.

    • If you modify the Name of target Topic, data will be written to the specified topic.

    Number of Partitions

    The number of partitions for the target topic.

    Prefixes of Keys to Be Synchronized

    For more information, see Set filter conditions.

    Prefixes of Keys to Be Filtered Out

    Additional Columns

    For more information, see Add additional columns.

  3. Click OK.

FAQ

Previous：Sync self-managed Redis on ECS to TairNext：Synchronize data from a TiDB database