DocsICP FilingConsole
官方文档
首页

Create a data subscription task for PolarDB for PostgreSQL (Compatible with Oracle)

更新时间：
复制 MD 格式
产品详情
我的收藏

This topic describes how to create a data subscription task for PolarDB for PostgreSQL (Compatible with Oracle).

Prerequisites

Usage notes

Type

Description

Source database limitations

  • Tables that you want to subscribe to must have a primary key or a unique constraint. All fields in the key or constraint must be unique. Otherwise, DTS might capture duplicate data.

  • If you subscribe at the table level, a single data subscription task supports a maximum of 500 tables. If you exceed this limit, the task fails when you submit it. In this case, you can split the tables across multiple subscription tasks or create a single task to subscribe to the entire database.

  • WAL logs:

    • You must enable WAL by setting the wal_level parameter to logical. This setting adds the information required to support logical decoding to the write-ahead logging (WAL). For more information, see Set cluster parameters.

    • The source database must retain WAL logs for more than 24 hours. If the retention period is shorter, DTS may fail to obtain the logs, which can lead to task failures or, in extreme cases, data inconsistency or loss. The DTS Service Level Agreement (SLA) does not cover issues caused by a WAL log retention period shorter than the required duration.

  • If the source instance is a read-only or temporary instance, ensure that it records transaction logs.

  • Due to limitations of logical replication on the source database, if a single row of data exceeds 256 MB after an update while the subscription instance is running, the subscription instance may fail and become unrecoverable. You must re-create the subscription instance.

Other limitations

  • DTS does not support subscribing to schema changes from the source instance.

  • A single data subscription task can subscribe to only one database. To subscribe to multiple databases, you must create a separate data subscription task for each database.

  • If a single subscribed row of data exceeds 16 MB, the consumer client cannot consume it. Attempting to consume it may cause an Out of Memory (OOM) error in the consumer client.

  • Verify that the subscription precision for columns of the FLOAT or DOUBLE data type meets your business requirements. DTS uses the ROUND(COLUMN,PRECISION) function to read values from these columns. If you do not specify a precision, DTS subscribes to FLOAT data with a precision of 38 digits and DOUBLE data with a precision of 308 digits.

  • DTS converts time-related data types that include a time zone to the UTC+8 time zone by default.

  • If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust its parameters.

    Note

    Parameter adjustments modify only DTS instance parameters and do not affect database parameters.

Procedure

  1. Use one of the following methods to go to the Change Tracking Tasks page and select the region in which the change tracking instance resides.

    DTS console

    1. Log on to the DTS console.

    2. In the left-side navigation pane, click Change Tracking.

    3. In the upper-left corner of the page, select the region in which the change tracking instance resides.

    DMS console

    Note

    The actual operations may vary based on the mode and layout of the DMS console. For more information, see Simple mode console and Customize the layout and style of the DMS console.

    1. Log on to the DMS console.

    2. In the top navigation bar, move the pointer over Data + AI > DTS (DTS) > Change Tracking.

    3. From the drop-down list to the right of Change Tracking Tasks, select the region in which the change tracking instance resides.

  2. Click Create Task to go to the task configuration page.

  3. Configure parameters in the Source Database and Consumer Network Type sections.

    Warning

    After you specify the source database instance, we recommend that you read the Limits that are displayed in the upper part of the page. Otherwise, the task may fail or the tracked data cannot be consumed.

    Type

    Parameter

    Description

    N/A

    Task Name

    The name of the change tracking task. DTS automatically assigns a name to the task. We recommend that you specify a descriptive name that makes it easy to identify the task. You do not need to use a unique task name.

    Source Database

    Select Existing Connection

    • If you want to use a database instance that is already registered with the system (newly created or saved), select it from the drop-down list. The system automatically fills in the database connection information.

      Note

      In the DMS console, this parameter is named Select a DMS database instance..

    • If your database instance is not registered, or you do not want to use a registered instance, you must manually configure the database connection information below.

    Database Type

    Select PolarDB (Compatible with Oracle).

    Access Method

    This topic uses Public IP Address as an example.

    Instance Region

    Select the region where the PolarDB for PostgreSQL (Compatible with Oracle) cluster is located.

    Data Sources

    Select Single Data Source. Then, in the Domain Name or IP Address field, enter the endpoint of the primary node or the cluster endpoint of the source PolarDB for PostgreSQL (Compatible with Oracle) cluster and specify the Port.

    Domain Name or IP Address

    Enter the endpoint of the primary node or the cluster endpoint.

    Port

    Enter the service port number of the source database. The default value is 1521.

    Database Name

    Enter the name of the database that you want to subscribe to.

    Database Account

    Enter a privileged account for the PolarDB for PostgreSQL (Compatible with Oracle) database.

    Database Password

    The password that is used to access the database instance.

    Consumer Network Type

    Network Type

    The value of the Network Type parameter is fixed to VPC. You must select a VPC and a vSwitch. For more information, see VPCs.

    Note

    • After a change tracking task is configured, you cannot change the settings in the Consumer Network Type section.

    • If your change tracking client is deployed in a VPC, we recommend that you select the same VPC and vSwitch connected to the client.

    • If you track data changes over internal networks, the network latency is minimal.

  4. In the lower part of the page, click Test Connectivity and Proceed.

    Note

    • Make sure that the CIDR blocks of DTS servers can be automatically or manually added to the security settings of the source and destination databases to allow access from DTS servers. For more information, see DTS server IP whitelist.

    • If the source database is a self-managed database and its Access Method is not set to Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

  5. Configure the objects for change tracking.

    1. On the Configure Objects page, configure the objects for change tracking.

      Parameter

      Description

      Data Change Types

      The Data Change Types checkbox is selected by default, and you cannot change it.

      The data change types that you want to track. You can select only Data Update. DTS tracks data updates of the objects that you select. Data updates include the INSERT, DELETE, and UPDATE operations.

      Source Objects

      Select one or more objects from the Source Objects section and click the Right arrow icon to add the objects to the Selected Objects section.

      Note

      You can select tables or databases as the objects for change tracking.

      • If you select a database as the object, DTS tracks incremental data of all objects, including new objects in the database.

      • If you select a table as the object, DTS tracks only incremental data of this table. In this case, if you want to track data changes of another table, you must add the table to the object list. For more information, see Modify the objects for change tracking.

    2. Click Next: Advanced Settings to configure advanced settings.

      Parameter

      Description

      Dedicated Cluster for Task Scheduling

      By default, DTS schedules the task to a shared cluster. You do not need to configure this parameter. You can purchase dedicated clusters of specified specifications to run DTS change tracking tasks. For more information, see What is a DTS dedicated cluster?.

      Retry Time for Failed Connections

      The retry time range for failed connections. If the change tracking task fails, DTS immediately retries a connection within the time range. Valid values: 10 to 1440. Unit: minutes. Default value: 720. We recommend that you set the parameter to a value greater than 30. If DTS is reconnected to the source database instance within the specified time range, DTS resumes the change tracking task. Otherwise, the change tracking task fails.

      Note

      • If multiple change tracking tasks are configured for a database instance, the shortest retry time range takes precedence. For example, Task A and Task B are configured for the same database instance. Task A is configured with a retry time range of 30 minutes, and Task B is configured with a retry time range of 60 minutes. In this case, the retry time range of 30 minutes takes precedence.

      • When DTS retries a connection, fees are charged. We recommend that you specify the retry time range based on your business requirements, or release the DTS instance at the earliest opportunity after the source database instance is released.

      Retry Time for Other Issues

      The retry time range for other issues. For example, if DDL or DML operations fail to be performed after the change tracking task is started, DTS immediately retries the operations within the retry time range. Valid values: 1 to 1440. Unit: minutes. Default value: 10. We recommend that you set the parameter to a value greater than 10. If the failed operations are successfully performed within the specified retry time range, DTS resumes the change tracking task. Otherwise, the change tracking task fails.

      Important

      The value of the Retry Time for Other Issues parameter must be smaller than the value of the Retry Time for Failed Connections parameter.

      Environment Tag

      You can select a tag to identify the environment of the instance. This parameter is optional.

      Monitoring and Alerting

      Specifies whether to enable alerting for the change tracking task. If alerting is configured and the task fails or the latency exceeds the threshold, alert notifications are sent. Valid values:

      • No: does not enable alerting.

      • Yes: enables alerting. In this case, you must also configure the alert threshold and alert contacts. For more information, see Configure monitoring and alerts.

  6. In the lower part of the page, click Next: Save Task Settings and Precheck.

    • To view the parameters to be specified when you call the relevant API operation to configure the DTS task, move the pointer over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

    • If you do not need to view or have viewed the parameters, click Next: Save Task Settings and Precheck in the lower part of the page.

    Note

    • Before you can start the change tracking task, DTS performs a precheck. You can start the change tracking task only after the task passes the precheck.

    • If the task fails to pass the precheck, click View Details next to each failed item. After you troubleshoot the issues based on the error message, you can run a precheck again.

    • If an alert is generated for an item during the precheck, perform the following operations based on the scenario:

      • If an alert item cannot be ignored, click View Details next to the failed item and troubleshoot the issues. Then, run a precheck again.

      • If the alert item can be ignored, click Confirm Alert Details. In the View Details dialog box, click Ignore. In the message that appears, click OK. Then, click Precheck Again to run a precheck again. If you ignore the alert item, data inconsistency may occur and your business may be exposed to potential risks.

  7. Purchase the instance.

    1. Wait until Success Rate becomes 100%. Then, click Next: Purchase Instance.

    2. On the Purchase page, specify the billing method of the change tracking instance. The following table describes the parameters.

      Parameter

      Description

      Billing method

      • Subscription: You pay for your subscription when you create an instance. The subscription billing method is more cost-effective than the pay-as-you-go billing method for long-term use. You are offered lower prices for longer subscription durations.

      • Pay-as-you-go: A pay-as-you-go instance is billed on an hourly basis. We recommend that you select the pay-as-you-go billing method for short-term use. If you no longer require a pay-as-you-go instance, you can release the instance to reduce costs.

      Resource Group Settings

      The resource group to which the instance belongs. Default value: default resource group. For more information, see What is Resource Management?

      Subscription Duration

      If you select the subscription billing method, set the subscription duration and the number of instances that you want to create. The subscription duration can be one to nine months, one year, two years, three years, or five years.

      Note

      This parameter is available only if you select the Subscription billing method.

    3. Read and select the Data Transmission Service (Pay-as-you-go) Service Terms.

    4. Click Buy and Start to start the change tracking task. You can view the progress of the task in the task list.

What's next

After the task is running, create consumer groups and connect your downstream clients to consume the tracked data.

  1. Create consumer groups

  2. Connect a client to consume the tracked data:

Previous：Create a PolarDB-X 1.0 change tracking taskNext：Create an Oracle data subscription task