Scheduled SQL Exactly-Once
Scheduled SQL in Simple Log Service writes job results to a destination Logstore or Metricstore with an Exactly-Once guarantee, preventing data duplication and loss. This is achieved through the idempotent writes feature of Logstore.
The following concepts apply to both Logstore and Metricstore. Logstore is used as the example for simplicity.
Background
A Logstore in Simple Log Service consists of multiple shards, each mapped to a half-open hash key range. You can write data to a Logstore by using either the load balancing method or the specified hash key mode. Simple Log Service routes data to a specific shard based on the hash key. For more information, see shard.
-
Load balancing method: Each data packet is randomly written to an available shard.
-
Specified hash key mode (idempotent writes): Data is written to the shard whose hash key range includes the specified hash key.
Idempotent writes
Idempotent writes in Simple Log Service require a hash key and a sequence id.
-
The hash key determines the destination shard for the data.
-
The sequence id identifies a write operation for a specific hash key.
A monotonically increasing sequence id ensures idempotency and prevents data duplication from retries.
-
If you write data with a sequence id less than or equal to the last recorded sequence id for the same hash key, Simple Log Service rejects the write and returns an error.
-
If you write data with a sequence id greater than the last recorded sequence id for the same hash key, Simple Log Service accepts the write and updates the sequence id for that key.
-
Scheduled SQL Exactly-Once with idempotent writes
Scheduled SQL uses idempotent writes to deliver data to the destination Logstore. To achieve Exactly-Once delivery, Scheduled SQL generates a unique hash key for each job and a monotonically increasing sequence id for each job instance.
-
Scheduled SQL assigns a unique hash key to each Scheduled SQL job.
When multiple Scheduled SQL jobs write to the same Logstore, each job receives a distinct hash key to prevent sequence id conflicts.
-
Scheduled SQL assigns a monotonically increasing sequence id to each instance within the same Scheduled SQL job.
If the following methods concurrently write to the same hash key in the same Logstore, a sequence id conflict occurs and one of the write operations fails, resulting in data loss.
-
Using an API or SDK to write data with explicit hash key and sequence id parameters.
-
Writing data using Logtail with its Exactly-Once feature enabled. For more information, see Logtail configurations (legacy).
-
Writing data using a Scheduled SQL job.