Adaptive message processing duration

Background concepts

• Visible: The message is available in the queue. Any consumer can receive and process it. A consumer is a service or application that receives and processes messages.

• Invisible: After a message is received by a single consumer, it becomes invisible and cannot be received by other consumers. This prevents other consumers from processing the same message.

• Visibility timeout period: The period from when a message is received by a consumer (invisible to other consumers) to when it can be received by other consumers again (visible to other consumers).

After a message is received by a single consumer, it becomes invisible and cannot be received by other consumers. The invisible status depends on visibility timeout period that prevents other consumers from processing the same message.

For more information about the visibility timeout period, see Visibility Timeout.

Scenarios

Simple Message Queue (formerly MNS) specifies a default visibility timeout period for each message. The countdown starts as soon as a consumer receives a message. If the consumer fails to process the message within the visibility timeout period, another consumer might receive and process it. This timeout is beneficial because messages must be explicitly deleted after processing. If an abnormal event occurs, such as a consumer process crash, the message can still be processed by another consumer.

Some users set a long default visibility timeout period for a queue to ensure that a message is not released before the consumer finishes processing it. For example, a queue's visibility timeout period is set to 6 hours. A consumer receives and processes message M1. If the process then crashes or the machine restarts, message M1 cannot be received and processed by another consumer for at least 6 hours. Writing custom code to handle these failures adds complexity to the program.

In scenarios that require high real-time performance and quick responses for every message, message consumption must meet the following goals:

• The queue's visibility timeout period is short. If a process crashes, another consumer can receive and process the unprocessed message after the short visibility timeout period expires.

• Messages must not time out during processing. For example, if a consumer takes more than 5 minutes to process a message, the message must not time out during this process.

Solution

The C# SDK provided by Simple Message Queue (formerly MNS) can handle this scenario. The solution is to periodically check whether a ChangeVisibility operation is needed for the message while the consumer is processing it. After the processing is complete, the consumer must still actively delete the message.

For the C# SDK download link, see C++ SDK. If you experience any issues with the demo, you can submit a ticket.

The following points apply to the program:

• Before you run the program, you must specify the accessId, accessKey, and EndPoint.

• Variable descriptions:

  Variable	Description
  MessageMinimalLife	The minimum life duration a message must have when it is registered. For example, if a message has only 0.1 seconds of its timeout period remaining when it is registered, there is not enough time to extend its life with `ChangeVisibility`. Therefore, `MessageMinimalLife` ensures that the message survives long enough for `ChangeVisibility` to be called. Set this value as needed based on your business workload.
  TimerInterval	The interval for the internal timer in the Manager. Ensure the timer starts before the message's remaining life reaches `MessageMinimalLife`. A short interval can be set, but this leads to more frequent checks.
  QueueMessageVisibilityTimeout	The default visibility timeout period for a message. This is a property of the queue. Each time `ChangeVisibility` is called, the message's visibility timeout period is reset to `QueueMessageVisibilityTimeout`. Therefore, its value must be greater than `TimerInterval` + `MessageMinimalLife` to ensure the message does not time out.
  MessageTimeout	The timeout period for a message within the Manager. For example, if a consumer gets stuck and a message is not processed after 5 hours (assuming 5 hours is much longer than the normal processing time), the Manager stops calling `ChangeVisibility` for the message. It lets the message's visibility time out.

• Flow description:

  • After a consumer receives a message, it first registers the message, then processes it, and finally calls the Manager's deleteMessage method.

  • When a message is first registered, the Manager uses the ThreadPool to schedule a `ChangeVisibilityTask` to check whether a `ChangeVisibility` operation is needed. The Manager also adds the message to its internal message list.

  • The Manager's internal timer periodically uses `Parallel` to start a `ChangeVisibilityTask` that checks all messages in the message list.

  • The specific operations related to Manager.ChangeMessageVisibility (ChangeVisibilityTask) are shown in the flowchart.

    The flowchart is as follows: