Lindorm Table supports real-time change tracking, which captures incremental data for you to consume in custom business scenarios.
How it works
The change tracking feature of Lindorm records every data change in a table. You can view these change records on your client. After you enable change tracking for a table, each insert, update, or delete generates a stream record. Each record is a key-value pair: the key is the row's primary key, and the value contains the pre-change value, post-change value, timestamp, and operation type.
Subscription modes
You can enable the Lindorm change tracking feature in two modes: pull and push.
-
Pull model: Subscribe to data directly through an open-source Kafka client. The service stores data temporarily. You must set an expiration time for the stored data, after which it is deleted. Create a data subscription channel in pull model.
-
Push model: The Lindorm change tracking feature pushes data directly to your Kafka client without server-side storage. Create a data subscription in push model.
Data subscription lifecycle
-
If you release the LTS cluster without stopping the task first, the sync task pauses and data cannot be consumed.
-
If change tracking is enabled but no data is consumed, the subscription is automatically canceled by default after 48 hours and the data is deleted.
Limits
-
The change tracking feature of Lindorm supports only at-least-once semantics. The same record may be consumed multiple times due to internal retries.
-
Incremental messages from the Lindorm change tracking feature are unordered. Message consumption order does not match the write order to the Lindorm wide table.
-
The change tracking feature for HBase tables requires Lindorm Tunnel Service (LTS) version 3.7.0 or later.
Note-
If LTS is already enabled, upgrade to the latest version before enabling change tracking. Minor version update.
-
If LTS is not enabled, you can purchase it when enabling change tracking. The latest version is purchased by default. Enable change tracking.
-
-
We recommend that you do not enable change tracking on non-SSD instances. Exporting OldImage or NewImage values adds an extra read per write to retrieve pre-update data. If you only need delta data,submit a ticket.
-
In pull model, subscribed data is stored for seven days by default and shares Lindorm Distributed File System (LDFS) storage space. Ensure sufficient storage before enabling change tracking.