Create a data subscription

Updated at:

Data Transmission Service (DTS) tracks data changes from ApsaraDB RDS for PostgreSQL instances using PostgreSQL's write-ahead logging (WAL) logical decoding mechanism. This topic describes how to create a change tracking task for an ApsaraDB RDS for PostgreSQL instance.

Prerequisites

Before you begin, make sure that you have:

Limits

Source database requirements

Requirement Details Impact if violated
Primary key or UNIQUE constraint Source tables must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Tracked changes may contain duplicates.
Table limit per task A single change tracking task supports up to 500 tables. To track more than 500 tables, create multiple tasks in batches or track the entire database instead. Exceeding this limit causes a request error.
WAL log retention WAL logs must be retained for more than 24 hours. If logs expire before DTS reads them, the task fails and data inconsistency or loss may occur, which can affect your Service Level Agreement (SLA).
Read-only or temporary instances If the source instance is a read-only or temporary instance, make sure the instance records transaction logs. The task cannot read changes if transaction logs are unavailable.
Primary/secondary switchover To perform a primary/secondary switchover without interrupting the task, enable the Logical Replication Slot Failover feature beforehand. For more information, see Logical Replication Slot Failover. Without this feature enabled, logical subscriptions are interrupted and the task stops.

Other limits

Limit Details
DDL tracking To track schema updates, create a trigger and a function in the source database to capture DDL information. For more information, see Use triggers and functions to implement incremental DDL migration for PostgreSQL databases.
Heartbeat table DTS adds a heartbeat table named dts_postgres_heartbeat to the source database to maintain accurate latency metrics.
Replication slot management DTS creates a replication slot prefixed with dts_sync_ for the source database. DTS clears historical replication slots every 90 minutes. If the task is released or fails, DTS clears the replication slot automatically. If a primary/secondary switchover is performed, log on to the secondary database to clear the replication slot manually. Replication slot information
Single database per task A single change tracking task tracks data from only one database. Create a separate task for each database.
FLOAT and DOUBLE precision DTS uses the ROUND(COLUMN,PRECISION) function to retrieve values from FLOAT and DOUBLE columns. The default precision is 38 digits for FLOAT and 308 digits for DOUBLE. Verify that these defaults meet your business requirements.
Warning

Do not change the endpoint or zone of the source ApsaraDB RDS for PostgreSQL instance while the change tracking task is running. Doing so causes the task to fail.

Create a change tracking task

Step 1: Go to the Change Tracking Tasks page

  1. Log on to the .

  2. In the top navigation bar, move the pointer over Data + AI.

  3. Choose DTS (DTS) > Change Tracking.

The navigation steps may vary depending on the mode and layout of the DMS console. For more information, see Simple mode and Customize the layout and style of the DMS console. You can also go directly to the Change Tracking page of the new DTS console.

Step 2: Configure the source database

  1. Click Create Task.

  2. (Optional) In the upper-right corner of the page, click New Configuration Page.

    Skip this step if the Back to Previous Version button is already displayed. The new configuration page is recommended when available.
  3. Read the Limits displayed at the top of the page before specifying the source database instance.

  4. Configure the source database and consumer network settings:

    Section Parameter Description
    (None) Task Name A name for the change tracking task. DTS assigns a default name. Specify a descriptive name to identify the task more easily. The name does not need to be unique.
    Source Database Select an existing database connection Select an existing instance to reuse its settings, or leave blank to configure the database manually.
    Database Type Select PostgreSQL.
    Access Method Select Alibaba Cloud Instance.
    Instance Region The region where the source ApsaraDB RDS for PostgreSQL instance resides.
    Instance ID The ID of the source instance.
    Database Name The name of the source database.
    Database Account The privileged account for the source database. The account must be the owner of the database. For more information about creating accounts and granting permissions, see Create an account and Create a database.
    Database Password The password for the database account.
    Consumer Network Type Network Type Fixed to VPC. Select a VPC and a vSwitch. This setting cannot be changed after the task is configured. If the change tracking client is deployed in a VPC, select the same VPC and vSwitch as the client to minimize network latency. For more information, see VPCs.
  5. Click Test Connectivity and Proceed. DTS automatically adds its CIDR blocks to the security settings of Alibaba Cloud database instances and ECS-hosted databases. For self-managed databases in data centers or third-party clouds, manually add the DTS CIDR blocks to the database security settings. For more information, see Add the CIDR blocks of DTS servers.

    Warning

    Adding DTS CIDR blocks to instance whitelists or ECS security group rules may introduce security risks. Before proceeding, take preventive measures such as: using strong credentials, restricting exposed ports, authenticating API calls, auditing whitelist rules regularly, or connecting through Express Connect, VPN Gateway, or Smart Access Gateway.

Step 3: Select objects and configure advanced settings

Configure the following settings under Basic Settings and Advanced Settings:

Basic settings

Parameter Description
Data Change Types Select the types of changes to track: Data Update tracks INSERT, DELETE, and UPDATE operations. Schema Update tracks create, delete, and modify operations on all object schemas. Use the change tracking client to filter tracked data. If you select Schema Update, create a trigger and a function in the source database first. For more information, see Use triggers and functions to implement incremental DDL migration for PostgreSQL databases.
Source Objects Select tables or databases from the Source Objects section, then click the 向右小箭头 icon to add them to the Selected Objects section. If you select a database, DTS tracks all objects in the database, including new objects added after the task starts. If you select a table, DTS tracks only that table. To add more tables later, see Modify the objects for change tracking.

Advanced settings

Parameter Description
Monitoring and Alerting Select Yes to configure alerts for task failures or latency spikes. Specify an alert threshold and alert contacts. Select No to skip alerting.
Retry Time for Failed Connections The time range during which DTS retries failed connections. Valid values: 10–1440 minutes. Default: 120 minutes. Set this to at least 30 minutes. If DTS reconnects within the retry window, the task resumes. If not, the task fails.
Note

If an instance serves as the source or destination database of multiple change tracking tasks, the shortest retry time range that is set for the instance takes precedence. During retries, you are charged for the DTS instance.

Configure ETL Select Yes to configure extract, transform, and load (ETL) rules using the code editor. Select No to skip ETL. For more information, see What is ETL?.

Step 4: Run the precheck

  1. Click Next: Save Task Settings and Precheck. To preview the API parameters used to configure this task, hover over the button and click Preview OpenAPI parameters.

  2. Wait for the precheck to complete. The task can start only after passing the precheck.

    • If a check item fails, click View Details next to the item, fix the issue, and click Precheck Again.

    • If a check item shows an alert that can be ignored, click Confirm Alert Details, click Ignore in the dialog box, click OK, and then click Precheck Again. Ignoring alert items may cause data inconsistency.

  3. When Success Rate reaches 100%, click Next: Purchase Instance.

Step 5: Purchase the instance

On the Purchase page, configure the billing settings:

Parameter Description
Billing method Subscription: Pay upfront for a fixed duration. More cost-effective for long-term use. Pay-as-you-go: Billed hourly. Suitable for short-term use. Release the instance when it's no longer needed to stop charges.
Resource Group Settings The resource group for the instance. Default: default resource group. For more information, see What is Resource Management?.
Subscription Duration (Subscription only) Set the subscription duration: 1–9 months, or 1, 2, 3, or 5 years.
  1. Read and select Data Transmission Service (Pay-as-you-go) Service Terms.

  2. Click Buy and Start. The task appears in the task list with its current progress.

What's next

When the change tracking task is running, create consumer groups and start consuming the tracked data.

  1. Create and manage consumer groups. For more information, see Create consumer groups.

  2. Consume the tracked data using one of the following methods: