Message accumulation

How to handle message accumulation?

In addition to providing asynchronous decoupling, message queues handle frontend traffic spikes to ensure backend system stability. This requires the message queue to have a certain capacity for message accumulation. A message queue can support the accumulation of billions of messages without a significant drop in performance.

Problem description

On the Consumer Status page for a Group ID in the Message Queue console, the real-time message accumulation is higher than expected, and performance has dropped significantly.

Solution

If you encounter message accumulation and a significant drop in performance, perform the following steps:

1. In the Message Queue console, go to the View consumer status page to find the host IP address of the consumer instance with the message accumulation. Then, log on to the host or container.

2. Run one of the following commands to view the process ID (PID).

   • ps -ef |grep java

   • jps -lm

3. Run the jstack -l pid > /tmp/pid.jstack command to view the stack information.

4. Run the cat /tmp/pid.jstack|grep ConsumeMessageThread -A 10 --color command to view information about ConsumeMessageThread. Focus on the thread state and stack. The command output is shown in the following figure.

   

For more information about thread states, see the official Java documentation.

More information

If the performance issue caused by message accumulation persists after you perform these steps, you can submit a ticket for technical support. When you submit the ticket, include the following information:

• The heap.bin file. You can obtain this file by running the jmap -dump:format=b,file=heap.bin [pid] command. Then, run the gzip heap.bin command to create a compressed package.

  Note

  [pid] is the process ID that you obtained in Step 2.

• The sofamq.log local log file from the consumer client where the message accumulation occurred.

• The version of the consumer client.