Overview of Real-Time Data Subscriptions
Lindorm Table supports real-time change tracking, which captures incremental data that you can consume to build custom business scenarios.
How it works
The change tracking feature of Lindorm records every data change in a table. You can view change records on your client. After you enable change tracking for a table, each insert, update, or delete generates a stream record. Each record is a key-value pair: the key is the row's primary key, and the value contains the pre-change value, post-change value, timestamp, and operation type.
Subscription modes
Lindorm change tracking supports two subscription modes: pull and push.
-
Pull model: Subscribe to data through an open-source Kafka client. The service stores data temporarily. You must set an expiration time for the stored data, after which it is deleted. Create a data subscription channel in pull model.
-
Push model: The Lindorm change tracking pushes data directly to your Kafka client without server-side storage. Create a data subscription in push model.
Data subscription lifecycle
-
If you release the LTS cluster without stopping the sync task first, the task pauses and data cannot be consumed.
-
If change tracking is enabled but no data is consumed, the subscription is automatically canceled after 48 hours by default, and the data is deleted.
Limits
-
Lindorm change tracking supports only at-least-once semantics. The same record may be consumed multiple times due to internal retries.
-
Incremental messages from Lindorm change tracking are unordered. The consumption order does not match the write order to the Lindorm wide table.
-
Change tracking for HBase tables requires Lindorm Tunnel Service (LTS) version 3.7.0 or later.
Note-
If LTS is already enabled, upgrade to the latest version before enabling change tracking. Minor version update.
-
If LTS is not enabled, you can purchase it when enabling change tracking. The latest version is purchased by default. Enable change tracking.
-
-
Do not enable change tracking on non-SSD instances. Exporting OldImage or NewImage values adds an extra read per write to retrieve pre-update data. If you only need delta data,submit a ticket.
-
In pull mode, subscribed data is stored for seven days by default and shares Lindorm Distributed File System (LDFS) storage space. Ensure sufficient storage before enabling change tracking.