Add a consumer group

更新时间:
复制 MD 格式

Change Tracking lets you create consumer groups. By creating multiple consumer groups, you can consume data from the same data source repeatedly, reducing the cost of using Change Tracking.

Prerequisites

A change tracking instance has been created, and its status is Normal. For more information, see Overview of change tracking scenarios.

Usage notes

  • If a single row of data to be consumed exceeds 16 MB, the consumer client may encounter an out of memory (OOM) error.

  • You can create a maximum of 20 consumer groups for a change tracking instance to allow for repeated data consumption.

  • Each consumer group supports only one consumer, and that consumer consumes all the data for the group.

  • If you switch the Data Transmission Service (DTS) incremental data collection module or the client, data duplication may occur.

Procedure

  1. Use one of the following methods to go to the Change Tracking Tasks page and select the region in which the change tracking instance resides.

    DTS console

    1. Log on to the DTS console.

    2. In the left-side navigation pane, click Change Tracking.

    3. In the upper-left corner of the page, select the region in which the change tracking instance resides.

    DMS console

    Note

    The actual operations may vary based on the mode and layout of the DMS console. For more information, see Simple mode console and Customize the layout and style of the DMS console.

    1. Log on to the DMS console.

    2. In the top navigation bar, move the pointer over Data + AI > DTS (DTS) > Change Tracking.

    3. From the drop-down list to the right of Change Tracking Tasks, select the region in which the change tracking instance resides.

  2. Find the target change tracking instance and click its instance ID.

  3. In the navigation pane on the left, click Consume Data.

  4. On the Consume Data page, click Add Consumer Group in the upper-left corner.

  5. In the Add Consumer Group dialog box, configure the following parameters.

    Parameter

    Description

    Consumer Group Name

    We recommend that you use a descriptive name for easier identification.

    Account

    Enter the account for the consumer group.

    • The account name can contain uppercase letters, lowercase letters, digits, and underscores (_).

    • The account name must be 1 to 16 characters long.

    Password

    Enter the password for the account.

    • The password must contain characters from at least two of the following categories: uppercase letters, lowercase letters, digits, and special characters.

    • The password must be 8 to 32 characters long.

    Confirm Password

    Enter the same password again.

  6. Click OK.

What to do next

After you add the consumer group, select a data consumption method based on the source database type and consumer client type.

Consumption method

Applicable database types

Description

Consume data from a PolarDB-X 1.0 instance by using an SDK

PolarDB-X 1.0, DMS LogicDB

This method is for change tracking instances where the source database is PolarDB-X 1.0 or DMS LogicDB.

Note

This is the only supported consumption method for change tracking instances where the source database is PolarDB-X 1.0 or DMS LogicDB.

Consume data by using an SDK

MySQL, PolarDB for MySQL, PostgreSQL, Oracle, PolarDB (Compatible with Oracle), PolarDB for PostgreSQL, PolarDB-X 2.0

This method is suitable for most scenarios.

Note

This method is recommended for all supported database types.

Consume data by using flink-dts-connector

This method is for scenarios using the Apache Flink stream processing framework.

Consume data by using a Kafka client

This method is for users who have their own Kafka client and can develop from source code.