Improve alert handling efficiency with ARMS alerting dashboards
This topic describes how to use the historical alert overview dashboard and the alert handling efficiency overview dashboard to improve alert handling efficiency.
Background information
When handling system alerts, companies often face the following pain points:
How can you view which alerts occurred in the system over a period and identify the most frequent alert types?
Alert configurations are not converged because multiple people configure alerts across different systems. How can you view only the alerts you care about and their handling status?
An on-call system is in place, but measuring the alert handling efficiency of on-call staff is difficult. How can you ensure the effectiveness of the on-call system?
A single service has multiple alerts configured across different systems, which makes it impossible to view alert handling efficiency or the service's Service-Level Agreement (SLA) by service.
Whether targeted system optimizations have reduced the proportion of alerts and the proportion of time with active alerts.
Daily Operations and Maintenance (O&M) teams typically use the following metrics to measure alert handling efficiency and service stability. By adding tags to alerts, you can measure handling efficiency by collecting statistics on these metrics across different dimensions. ARMS also automatically associates the person handling an alert with that alert. This lets you collect statistics on how different people handle alerts.
MTTx
MTTD: Mean Time to Detect. The average time from when an alert occurs to when the alert management system receives it.
MTTA: Mean Time to Acknowledge. The average time from when an alert occurs to when it is first acknowledged.
MTTR: Mean Time to Recover. The average time from when an alert occurs to when it is recovered or closed.
Service Availability (SLA)
The formula for service availability is: Availability = System Uptime / (System Uptime + System Downtime)
You cannot calculate service availability from a single alert. To calculate the service availability metric for a type of alert, you must sum the durations of all alerts of that type. The ARMS alert management system uses a set of tags to identify a unique alert. Alerts with the same tags are considered the same alert. ARMS automatically removes duplicates and accumulates the duration for continuous alerts. Alerts that occur at different times are treated as separate alerts, but they are grouped into the same alert type. Within a fixed time window, the availability for a single alert type is calculated as 
. If multiple alert types are involved, the final availability metric on the dashboard is 
.
ARMS alerting dashboards
When you use the ARMS alert management system to centrally collect and handle alerts from various systems, it automatically gathers data about the alert lifecycle and user actions. This data is then visualized on Grafana dashboards. These dashboards help you understand alert distribution and handling, which improves system stability.
Historical alert overview dashboard
In the ARMS console, go to the page and click the Historical Alert Overview tab to view the historical alert overview dashboard.
On the Historical Alert Overview page, you can filter by time to view the alert distribution. You can also view MTTx and SLA metrics by any tag dimension.
View alert distribution by filtering by time
In the upper-right corner of the Historical Alert Overview page, select a time range to view the alert distribution for that period.
By default, alert distribution statistics are based on the alert name (labels_alertname). You can edit the Daily Alert Trend panel to change the dimension for the statistics.
For example, to view alert distribution statistics by alert severity, perform the following steps:
Click the name of the Daily Alert Trend panel, and then click Edit.
Modify the Prometheus Query Language (PromQL) query statement. Replace the alert name (labels_alertname) with the alert severity (labels_severity).
View MTTx and SLA metrics by any tag dimension
View the tags that are supported by the alert event.
On the page, click the name of the target event.
All fields in the Event Details panel in the Extended Fields section can be used as filter tags.
On the Historical Alert Overview tab of the Alert Overview page, set the tag key and value.
For example:
Query for all alert events where `clustername` is `cmonitor-online-test-shanghai`.
Query for alert events where `clustername` is `cmonitor-online-test-shanghai` and `severity` is `default`.
The filtered results show that the SLA is 99.88288% and the number of handled alerts (handler) is 1.083K. The Daily Alert Trend bar chart on the right shows the alert distribution from 04/09 to 04/15, in which error rate alerts total 608.
Alert handling efficiency overview dashboard
In the ARMS console, go to the page and click the Alert Handling Efficiency Overview tab to view the alert handling efficiency overview dashboard.
The alert handling efficiency overview dashboard displays the handling efficiency metrics for each person in the left section. The upper-right section displays the handling metrics for each service. The lower-right section displays the alert details.
View MTTx metrics for alert handlers:
Only alerts that are handled through DingTalk, WeCom, or Lark can be associated with specific people. This allows for the collection of alert metrics for each person. For more information, see Handle alerts in an alert notification group.
View service-level MTTx metrics for alerts:
In practice, a single service often has multiple alerts configured. The ARMS alert management system supports merging these alerts to calculate service-level alert statistics. ARMS adds business tags to alerts using event processing streams. For alerts that have business tags, ARMS automatically collects statistics at the service level. For more information, see Event processing streams.