Consumer group monitoring and alerts
Log Service provides monitoring features and a built-in alert mechanism for consumer group latency when you use consumer groups to consume data. You can view various reports, including consumer group lists, latency status, and data consumption ratios. You can also customize alert rules for consumer group latency monitoring. If consumption latency exceeds a specified threshold, Log Service automatically triggers an alert.
Prerequisites
The Important Logs feature is enabled for service logs. For more information, see Manage service logs.
Background
A consumer group consists of multiple consumers. Each consumer consumes data from a subset of shards in a Logstore. Consumers within the same consumer group work together to consume log data from a Logstore, ensuring that no data is consumed more than once.
The data model of a shard resembles a queue. New data is appended to the end of the queue, and each record is associated with a write timestamp. The following figure illustrates the shard data model.
View reports
The Consumer group monitoring dashboard displays metrics for consumer groups, such as their total count, associated Logstores and shards, latency, and data consumption ratios. It also provides a complete list of groups, a ranking of the top 10 by latency, and consumption latency charts. For more information, see View data reports.
Procedure
- Log on to the Log Service console.
-
In the Log Application area, on the Cloud Product Lens tab, click CloudLens for SLS.
-
In the upper-left corner of the page, select the target project.
-
In the left-side navigation pane, under Dashboard, click Access Monitoring.
-
Click Consumer group monitoring.
The Consumer group monitoring dashboard displays metrics for consumer groups, such as their total count, associated Logstores and shards, latency, and data consumption ratios. It also provides a complete list of groups, a ranking of the top 10 by latency, and consumption latency charts.
Set up alerts
Key concepts for consumption latency alerts:
-
Consumption process: Sequentially reading data from the head of the queue.
-
Consumption progress: The write timestamp of the data record that a consumer is currently reading.
-
consumption latency: The time difference, in seconds, between the current consumption progress and the write timestamp of the latest record in the queue.
The consumption latency of a consumer group is the maximum consumption latency among all its shards. If this value exceeds a specified threshold, it indicates a consumption lag and triggers an alert.
Procedure
- Log on to the Log Service console.
-
In the Log Application area, on the Cloud Product Lens tab, click CloudLens for SLS.
-
In the upper-left corner of the page, select the target project.
-
In the left-side navigation pane, click Alerts. On the Alert Rules tab, click .
-
In the Create from Template panel, click Consumer Group Latency Monitoring, and then configure the alert parameters. For more information about the parameters, see Create alert monitoring rules.