Create a CDC stream in push mode-Lindorm(Lindorm)-阿里云帮助中心

This topic describes how to use Lindorm streams to implement data subscription in Push mode. After you create a Lindorm stream for a table in your Lindorm database, the change tracking feature is enabled for the table. The change tracking feature records each data insert, update, and delete operation that is performed on the table. Incremental data involved in the recorded operations is pushed to a Message Queue for Apache Kafka topic that you specify when you create the Lindorm stream. Then, you can consume and use the incremental data to build or implement business services based on your requirements.

Process

The following figure shows how the messages of incremental data in a Lindorm table are pushed to Message Queue for Apache Kafka.

Prerequisites

The IP addresses of your Lindorm client and Message Queue for Apache Kafka client are added to the whitelist of your Lindorm instance. For more information, see Configure a whitelist.
The source Lindorm instance and the destination Message Queue for Apache Kafka instance are connected to Lindorm Tunnel Service (LTS). For more information, see Establish network connections.
A LindormTable data source is created. For more information, see Add a LindormTable data source.
A Kafka data source is created. For more information, see Add a Kafka data source.
The change tracking feature is enabled. For more information, see Enable change tracking.

Create a Lindorm stream

Go to the Lindorm Tunnel Service (LTS) console. In the left-side navigation pane, choose Change Tracking > Push.

Click create and configure the parameters.

Parameter	Description
Lindorm Cluster	Select the created LindormTable data source.
Table Name	The name of the table for which you want to capture data changes. The format is `namespace.tablename`. For example, `ns1.table1` specifies that you want to capture data from the `table1` table in the `ns1` namespace.
Blacklist table (Optional)	Specify tables whose incremental data should not be pushed. Changes in these tables are ignored.
MessageStorage Type	Select KAFKA.
Storage Datasource	Select the created Kafka data source.
MessageStorage Config	`kafka_topic`: Specify the name of the Kafka topic. `kafka_ttl`: Leave this empty. `kafka_partition_num`: Leave this empty. Important In push mode, you must create the topic in Kafka in advance.
MessageVersion	The message format. The default is DebeziumV2.
Message Config	`old_image`: Specifies whether the message includes the row values before a change. You must set this parameter to `true`. Setting it to `false` is not supported. `new_image`: Specifies whether the message includes the row values after a change. You must set this parameter to `true`. Setting it to `false` is not supported. `with_schema`: Specifies whether the message includes the table schema. Set this parameter to `false` to prevent oversized messages. `ignore_family_prefix`: Specifies whether to remove the column family prefix from the exported column names. For example, if the full column name is `f:name`, setting this parameter to `true` results in the exported column name `name`.

Click Submit.