Notification policy best practices
ARMS Alert Management lets you use a notification policy to route, group, compress, and aggregate alert events. You can then send notifications through various channels, including SMS messages, phone calls, emails, DingTalk, WeCom, Lark, and webhooks. In addition, notification policies support advanced features like repeat notifications, escalation policies, and on-call schedules for complex notification needs.
Background information
Generating a large number of alert events from alert rules or integrations requires timely notifications. However, managing a high volume of alerts can be challenging, as notification scenarios and methods vary. To solve these issues, ARMS provides notification policies that match alert events based on your criteria, compress and group them into alerts, and deliver notifications through a variety of channels. A notification policy also supports features like closing, silencing, and claiming alerts, to help you manage alerts efficiently and receive accurate, timely notifications.
How a notification policy works
To use a notification policy, you first need a source for alert events. ARMS supports two sources for alert events:
ARMS alert rules: Alert Management integrates with ARMS alert rules by default. In the ARMS console, you can configure alert rules for Application Monitoring, Browser Monitoring, Prometheus Monitoring, Kubernetes Monitoring, and Application Security. When the conditions of an alert rule are met, it generates an alert event that a notification policy can then match.
Integrations: Alert Management supports integrations with third-party products. You can send events from services like a self-managed Prometheus instance, Log Service (SLS), CloudMonitor, Grafana, Zabbix, and Skywalking to ARMS for centralized management. You can also use a custom integration to report events to ARMS. After you configure an integration, a notification policy can match events from that source.
After you configure a source for alert events, the notification policy processes the events in three stages: event matching, event processing, and alert notification. The following diagram shows the workflow.
Alert event matching
An alert event is the basic unit that a notification policy processes. Alert events are generated by ARMS alert rules or integrations. Each event has required basic properties and extended fields, which are key-value pairs called labels.
An alert event includes basic fields such as event name, description, start time, and end time, as well as extended fields. If all labels of events other than the basic fields are identical, ARMS automatically compresses them into a single event.
When you create a notification policy, you can match alert events by any label. The event matching rule supports various operators, such as equals and regular expression matching, to filter alert events.
Keep the following points in mind when you use an event matching rule:
An event matching rule can match all non-silenced alert events that meet its conditions.
Multiple conditions within a single event matching rule are combined with an AND operator. An alert event must satisfy all conditions to match the rule.
Multiple event matching rules are combined with an OR operator. An alert event that satisfies any of the rules is considered a match.
By defining event matching rules, you can accurately identify any alert event and configure it to be either notified or silenced based on your needs.
Event processing
A notification policy processes only the alert events that satisfy its event matching rule. You can configure how alert events are processed when you set up a notification policy.
Two event processing methods are supported:
Aggregate: Compresses matched alert events into a single alert or into multiple alerts by group.
When you choose to aggregate events, you can further group them by specified labels. Events are then compressed into multiple alerts based on these groups. Each group has its own lifecycle for triggering, sending, and resolving alerts, allowing for independent notification, resolution, and management.
After you configure grouping, if no new alert events are triggered for a group within five consecutive minutes, the system considers the alert resolved. When all alert events in a group are resolved, the system marks the alert as recovered and sends a recovery notification.
If you do not need recovery notifications, you can disable them. In this case, the event and alert will still be automatically resolved, but no notification will be sent upon recovery.
Silence: Marks all matched alert events as silenced, which prevents other notification policies from matching them.
When you set the processing method to Silence, you can specify a time period during which the silence rule is active. During this period, ARMS marks all alert events that meet the conditions as silenced, and no other notification policy can match them. When you select Silence as the processing method, you do not need to configure alert notification settings.
Alert notification
When the event processing method is set to Aggregate, ARMS compresses the alert events into one or more alert event groups based on the labels you specify. Each group of alert events becomes a single alert. In the notification policy, you can set the recipients and notification methods for the alert.
You can select a contact, contact group, DingTalk group, webhook, or on-call schedule as the recipient and choose the desired notification method.
You can choose one of the following escalation methods:
Repeat Notification: Resends a notification at a specified interval as long as the alert remains unresolved.
Escalation Policy: Uses a predefined escalation policy to notify different recipients through other channels if an alert remains unresolved.
Example scenarios
Follow these best practices when using notification policies:
Configure appropriate alert rules or integrations to generate alert events that contain all necessary fields.
Configure one or more notification policies to silence events as needed. For events that require long-term silencing, we recommend disabling the corresponding alert rule or stopping the integration from sending the event.
In general, create one notification policy for each distinct recipient group, and then set up an event matching rule to filter the events that you want to send to that group.
If you use an IM tool like DingTalk, WeCom, or Lark to receive alerts, we recommend using an IM chatbot to send the notifications. This allows you to use the action buttons on the notification card to silence, claim, and resolve alerts from any platform. You can also review alert generation data, statistics, resolution details, and mean time to resolution (MTTR) later.
The following three scenarios demonstrate how to use notification policies.
These three common scenarios show practical ways to use ARMS Alert Management for your notification needs. You can follow one of these scenarios or combine them to handle more complex business scenarios.
Scenario 1: Handle individual alerts
Configuring an alert rule to notify an individual is a basic and common use case.
Configure an alert rule
If you use ARMS Prometheus Monitoring, you can configure basic alert rules for container CPU and memory usage in the ARMS console.
If you do not use ARMS Prometheus Monitoring, you can report alert events through an integration and then view them on the Alert Event History page.
Configure a notification policy
After you generate an alert event or expect one to be generated, you can create a notification policy to process it, generate an alert, and send it to yourself.
Using the Prometheus alert rule from the previous example, if you want to send alert events from the current cluster to a DingTalk group, you first need to create a DingTalk chatbot and then create a notification policy.
Configure the notification policy as follows:
Event matching rule: To match all alert events generated by the two alert rules in the alert-dev cluster and avoid matching events from other rules, configure the rule to match alert events where the Cluster Name is alert-dev and the Alert Name is Container CPU Alert-Test or Container Memory Alert-Test.
Event handling: Set the processing method to Grouping. In this scenario, the matched alert events are generated by two alert rules. If you want events from each rule to trigger a separate alert with an independent notification cycle, set the event grouping fields to Cluster name and Alert name. This configuration sorts the events generated by the two alert rules into two separate alerts.
Alert Generation: For the recipient, select the DingTalk chatbot you created earlier. For the notification method, select DingTalk. Set the escalation method to Repeat Notification and configure it to resend the notification every 5 minutes as long as the alert is active.
After you configure the notification policy, click Save. You will then see the alert notifications in the corresponding DingTalk group.
Claim and resolve alerts
When you receive an alert notification in your DingTalk group, you can click the buttons on the card to handle the alert. The first time you handle an alert, you must bind your mobile number. For more information, see Handle alerts in an alert notification group.
Claim Alert: Sets the alert status to claimed, informing other users that someone is handling it.
Close/Resolve Alert: Sets the alert status to resolved and allows you to add a solution. If the underlying alert event is still active, the system generates a new alert with the same name.
Follow Alert: After you follow an alert, you can view it in the followed alerts list on the homepage of the side panel.
Push Alert: If you have integrated a ticketing system, you can push the alert to create a ticket.
Silence Alert: If an alert does not require immediate attention, you can silence notifications for a specific period.
Unresolved Alerts: Allows you to view a list of unresolved alerts.
Scenario 2: Route alerts to business teams
An SRE team might need to configure alert rules that route alerts to different business teams based on specific business labels.
Imagine a company with an SRE team and three business teams: A, B, and C. Their applications are deployed across three clusters: cluster-dev, cluster-prod-1, and cluster-prod-2. cluster-dev is a test cluster, while the other two are production clusters. Each business team's applications are deployed in a separate namespace.
The SRE team needs to configure basic alert rules for container CPU, memory, and disk usage, as well as some business-specific alert rules for each cluster. The notification requirements are as follows:
Alerts from the test cluster
cluster-devshould be sent to the corresponding business team's DingTalk group with a low notification frequency. No SMS messages or phone calls should be sent. Each business team should only receive alerts for the pods they are responsible for.All alerts from the production clusters
cluster-prod-1andcluster-prod-2must be sent to the SRE team. In addition, each business team should receive alerts for their own pods. These notifications should have a high frequency.
Use Prometheus alert rule templates to create alert rules
If your clusters are integrated with ARMS Prometheus Monitoring, you can follow these steps to create alert rules by using Prometheus alert rule templates. If you send events to Alert Management through an integration, you must configure the alert rules yourself.
In the ARMS console, on the page, first create an alert rule template based on your requirements. For more information, see Prometheus Alert Rule Templates.
After you create an alert rule template, apply it in batches to cluster-dev, cluster-prod-1, and cluster-prod-2.
On the page, you can see that the alert rules have been created in a batch, and four alert rules have been applied to each cluster.
Create contacts, contact groups, and IM chatbots
Add contacts for the SRE team and business teams A, B, and C. Notification channels can include contacts (for SMS, phone calls, and email), IM chatbots (for DingTalk, WeCom, and Lark), and webhooks. In this scenario, you must add contacts and a DingTalk group for each business team.
Create notification policies for alert notification
Requirement 1: Alerts from the test cluster cluster-dev should be sent to the corresponding business team's DingTalk group with a low notification frequency. No SMS messages or phone calls should be sent. Each business team should only receive alerts for the pods they are responsible for.
To meet this requirement, you need to create three notification policies, one for each business team's DingTalk group.
The notification policy for Team A's test environment is configured as follows:
Event Matching Rule: This policy needs to filter alert events from the test cluster that belong to pods managed by Team A. In this scenario, teams are isolated by namespace, so you can set the event matching rule as shown in the figure below. If your business teams are isolated by pod name or other custom labels, you can add extra filtering conditions.
Event Processing: Select Aggregate. Since alert events may come from multiple alert rules, group them by alert name. The cluster name is optional and does not affect the result.
Alert Generation: Since test environment alerts only need to go to the DingTalk group, select the DingTalk group and set the repeat notification frequency to once every 30 minutes.
Requirement 2: All alerts from the production clusters cluster-prod-1 and cluster-prod-2 must be sent to the SRE team. In addition, each business team should receive alerts for their own pods. These notifications should have a high frequency.
To meet this requirement, you need to create another four notification policies: one for the SRE team and one for each of the three business teams' production environments.
The SRE team needs to receive all alerts, so their notification policy is configured as follows:
Event Matching Rule: All alert events from the production clusters
cluster-prod-1andcluster-prod-2need to be sent to the SRE team. Therefore, you only need to filter by cluster name and add no other conditions.Event Processing: Since events may match from multiple clusters and alert rules, group them by cluster name and alert name.
Alert Generation: For the production environment, you need to notify SRE team members via SMS, phone, and their DingTalk group. Select the corresponding contact group and DingTalk group, choose the notification methods, and set the escalation method to repeat the notification every 5 minutes while the alert is active.
Each business team needs to receive alerts related to their own pods. For example, the notification policy for Team A is configured as follows:
Event matching rule: All alert events from the production clusters cluster-prod-1 and cluster-prod-2 that are related to Business Team A must be sent to Business Team A. In this scenario, the business team's applications are in different namespaces, so a filter condition is added where the namespace is equal to team-a.
Event Processing: Since events may match from multiple clusters and alert rules, group them by cluster name and alert name.
Alert Generation: For the production environment, you need to notify Team A members via SMS, phone, and their DingTalk group. Select the corresponding contact group and DingTalk group, choose the notification methods, and set the repeat notification frequency to every 5 minutes while the alert is active.
Following the example for Team A, create corresponding notification policies for Team B and Team C.
Once created, these policies will meet the notification requirements. For information on managing alerts in the DingTalk group, see Scenario 1 above.
In this scenario, you created a total of:
Four alert rule templates.
Several contacts and DingTalk chatbots.
Seven notification policies.
This setup allows you to monitor basic resources in both test and production environments and route alerts to different teams based on the environment and team responsibilities. If you need to add new alert rules in the future, you only need to add the rules without changing the notification policies. If team or notification requirements change, you only need to modify the notification policies.
Scenario 3: Notify based on schedules and escalations
In some cases, you may need to send alert notifications to different people based on an on-call schedule and escalate the notification if the primary recipient does not respond promptly.
For example, three team members—A, B, and C—cover a department's work hours on a rotating basis, while member D handles after-hours support. If the on-call person does not respond, the system must escalate the alert.
The specific requirements are:
Members A, B, and C are each on-call for two days in rotation. They only receive alerts between 9:00 AM and 6:00 PM daily. At all other times, member D is on call.
By default, alerts are sent only as SMS messages. However, if no one claims an alert within 10 minutes, the system calls the on-call person.
If the alert remains unclaimed 10 minutes after the phone call, the system calls the team lead.
Create contacts and an on-call schedule
Create contacts for members A, B, C, and D, as well as a DingTalk group for on-call alert notifications. For more information, see Contacts and DingTalk chatbot.
Create an on-call schedule.
Create Shift 1: Includes members A, B, and C. They receive alerts from 9:00 to 18:00 daily, with a rotation every 48 hours.
Create Shift 2: Includes member D. They receive alerts from 18:00 to 9:00 the next day, with no rotation.
Click Save to view the created on-call schedule.
Create an escalation policy
As required, if no one claims an alert within 10 minutes, the system must call the on-call person. If the alert remains unclaimed 20 minutes after the initial trigger, the system must call the team lead.
Create a notification policy
You can configure the event matching rule and event processing as described in Scenario 1 or Scenario 2. To use the on-call schedule and escalation policy you just configured, select the created on-call schedule for the Recipient/Schedule and the corresponding escalation policy for the Escalation Method.