Consumption idempotence

What is consumption idempotence?

An operation is idempotent when running it multiple times produces the same result as running it once. In a message queue, network exceptions or system failures can cause the same message to be delivered more than once. Idempotent consumption ensures your business logic executes exactly once even when the same message arrives repeatedly, preventing duplicate operations such as charging a customer twice.

Example: A consumer receives a payment deduction message for an order totaling USD 100. Due to network exceptions or system failures, SMQ re-delivers the message, and the consumer receives it repeatedly. With idempotent consumption in place, the payment is deducted only once and a single deduction record is written, regardless of how many times the message arrives.

When duplicate delivery happens

SMQ uses at-least-once delivery semantics. In distributed systems, any of the following can cause the same message to be delivered more than once:

• Processing time exceeds the invisible duration: If consumer A takes longer to process a message than the InvisibleDuration setting, SMQ assumes the message was not processed and re-delivers it to consumer B. Both consumers then process the same message.

• Network failure after successful processing: After a consumer finishes processing a message, a network failure prevents the DeleteMessage call from reaching SMQ. When the network recovers, SMQ re-delivers the message because it was never deleted.

• Broker or consumer restart: If a broker or consumer restarts after processing a message but before DeleteMessage completes, SMQ may re-deliver the message once the invisible duration expires.

Implement idempotent consumption

Choose an idempotent key

Do not use the message ID as your idempotent key. Messages with different message IDs may contain the same content, and when a client retransmits a message, the retransmission may be assigned a new ID. Relying on message IDs would cause your deduplication logic to miss the duplicate deliveries it is meant to catch.

Instead, derive your idempotent key from a stable business attribute that uniquely identifies the operation. For example:

• Order payment: {orderId}/{operationType}, such as order-9527/payment

• Inventory update: {itemSku}/{warehouseId}/{batchId}

The key must remain the same across all deliveries of the same logical message and must differ for genuinely distinct operations.

Deduplication patterns

Choose one or more of the following patterns based on your system's architecture:

• Unique identifier tracking: Assign a unique identifier to each message. When a message is delivered to a consumer, record the unique identifier of the message. This helps ensure that the message will not be consumed again by this consumer.

• Database unique constraint: Add a unique constraint on the idempotent key column in your database. When a duplicate message arrives and your code attempts to insert the same key, the database rejects the insert with a constraint violation. Catch this error and treat it as a successful no-op. To avoid a race condition between checking and inserting, wrap the check and insert in a single transaction or use an upsert operation.

• Business state check: Before executing any logic, query the current state of the message. Check whether the message exists or has already been processed. If so, skip processing and return immediately. This pattern works well when your business operations are naturally state-driven.

• Operation merging: For commutative or accumulative operations, design the operation so that applying it multiple times yields the same result as applying it once—for example, using SET balance = X instead of SET balance = balance - Y.