By configuring Prometheus alerting rules, you can define conditions for specific monitoring metrics. When these conditions are met, the system generates a corresponding alert event. To receive these alert notifications, you must configure a notification policy, which sends alerts to you through various channels such as SMS, email, phone calls, DingTalk group chatbots, WeChat Work chatbots, or Webhooks.
Prerequisites
Your data source is connected to Prometheus. For more information, see one of the following topics:
Procedure
Log on to the Managed Service for Prometheus console.
In the left-side navigation pane, click Alert Rule List.
On the Prometheus Alert Rules page, click Create Prometheus Alert Rule.
Create a Prometheus alert rule by using a static threshold
The static threshold check type provides a list of predefined alert metrics. This allows you to quickly create an alert rule for common metrics without writing a PromQL query.
On the Create Prometheus Alert Rule page, configure the following parameters.
Parameter
Description
Example
Alert name
A descriptive name for the alert rule.
prod-cluster-container-cpu-alert
Check type
Select static threshold.
static threshold
Prometheus instance
Select the Prometheus instance for which you want to create an alert.
prod-cluster
Contact group
Select a contact group.
Different Prometheus types support different contact groups. The available contact group options change depending on the selected Prometheus instance type.
Kubernetes Workloads
Alert metric
Select the alert metric to monitor. Each contact group has a different set of available metrics.
Container CPU Usage
Alert condition
Define the condition that triggers an alert event based on the selected metric.
The alert fires when Container CPU Usage is
Greater than80%.Filter condition
Set the scope for the alert rule. An alert event is generated only when a resource that matches the filter condition also meets the alert condition.
The available filter conditions are:
Traverse: The rule applies to all resources in the selected Prometheus instance. This is the default setting.
Equal To: Applies the rule to a single, specified resource. You cannot specify multiple resources.
Not Equal To: Applies the rule to all resources except for the one specified. You cannot specify multiple resources.
Match Regular Expression: Applies the rule to all resources whose names match the specified regular expression.
Do Not Match Regular Expression: Excludes all resources whose names match the specified regular expression from the alert rule.
NoteThe data preview section appears after you set a filter condition.
The filter condition cannot exceed 300 characters.
Traverse
Data preview
The data preview section displays the PromQL statement for the alert condition and shows the values of the monitored metric on a time series chart.
By default, only the real-time value for a single resource is shown. You can use the filters in this section to view values for different resources and time ranges.
NoteThe alert threshold is displayed as a red horizontal line on the time series chart. Chart lines that meet the alert condition are dark red, and lines that do not are blue.
Hover over a point on the chart to view resource details for that specific time.
Click and drag on the chart to zoom in on a specific time range.
N/A
Duration
An alert event is generated immediately if the condition is met even once.
An alert event is generated only if the condition is continuously met for a specified duration (N minutes).
1
Alert level
Set a custom alert level. The default level is Default. The severity increases from Default, P4, P3, P2, to P1.
Default
Alert message
The content of the alert notification that users receive. You can use Go template syntax to include variables in the alert message.
Namespace: {{$labels.namespace}} / Pod: {{$labels.pod_name}} / Container: {{$labels.container}} CPU usage {{$labels.metrics_params_opt_label_value}} {{$labels.metrics_params_value}}%, current value {{ printf "%.2f" $value }}%
Alert notification
Simple Mode: You can set the Notification Objects, Notification Period, and Repeat Policy.
Standard Mode:
Do Not Specify Notification Policy: If you select this option, you can create a notification policy on the Notification Policy page after you create the alert rule. You can specify matching rules and conditions, such as the alert rule name, to match this rule. When an alert is triggered, the alert message is sent to the specified contact or contact group in the notification policy. For more information, see Notification policies.
Select a notification policy: If you select this option, ARMS adds a matching rule to the policy. This rule uses the alert rule ID to ensure the policy matches events from the current alert rule.
ImportantSpecifying a notification policy only ensures that alert events from this rule are matched by the selected policy. However, these events might also be matched by other notification policies that use fuzzy matching. The relationship between alert events and notification policies is many-to-many.
Do not specify notification policy
Advanced settings
Check interval
The interval at which the alert rule is evaluated. The default and minimum value is 1 minute.
1
Check when data is complete
Yes
No
Yes
Tag
Set tags for the alert. These tags can be used as matching conditions in notification policies.
N/A
Annotation
Set annotations for the alert.
N/A
After you configure the parameters, click Save. On the Prometheus page, you can view the status of the new alert rule.
If the Status of the alert rule is Automatic Interruption, edit the rule based on the reason provided, and then click Start. In the dialog box that appears, click OK. If you cannot resolve the issue, contact support by using the ARMS alert service DingTalk ID ARMS.
An alert rule might be automatically interrupted for the following reasons:
The query returns more than 1,500 time series.
No notification object is configured in the alert management settings.
The Prometheus instance is uninstalled or unavailable.
Create a custom PromQL alert rule
To monitor a metric not available in the static threshold list, use the custom PromQL check type to create an alert rule.
On the Create Prometheus Alert Rule page, configure the following parameters.
Parameter
Description
Example
Alert name
A descriptive name for the alert rule.
Pod CPU usage greater than 8%
Check type
Select custom PromQL.
custom PromQL
Prometheus instance
Select the Prometheus instance for which you want to create an alert.
N/A
Reference alert contact group
Select a contact group.
Different Prometheus types support different contact groups. The available contact group options change depending on the selected Prometheus instance type.
Kubernetes Workloads
Reference metrics
Optional. The reference metrics include custom PromQL configurations for common metrics. You can select a similar metric to pre-fill the query and then modify it to meet your needs.
The available reference metrics are filtered based on the selected Prometheus type.
Pod disk usage alert
Custom PromQL statements
Use a PromQL expression to define the alert rule.
Namespace: {{$labels.namespace}}/Pod: {{$labels.pod_name}}/Disk device: {{$labels.device}} usage exceeds 90%, current value {{ printf "%.2f" $value }}%max(container_fs_usage_bytes{pod!="", namespace!="arms-prom",namespace!="monitoring"}) by (pod_name, namespace, device)/max(container_fs_limit_bytes{pod!=""}) by (pod_name,namespace, device) * 100 > 90
Data preview
The data preview section displays the PromQL statement for the alert condition and shows the values of the monitored metric on a time series chart.
By default, only the real-time value for a single resource is shown. You can use the filters in this section to view values for different resources and time ranges.
NoteHover over a point on the chart to view resource details for that specific time.
Click and drag on the chart to zoom in on a specific time range.
N/A
Duration
An alert event is generated immediately if the condition is met even once.
An alert event is generated only if the condition is continuously met for a specified duration (N minutes).
1
Alert level
Set a custom alert level. The default level is Default. The severity increases from Default, P4, P3, P2, to P1.
Default
Alert message
The content of the alert notification that users receive. You can use Go template syntax to include variables in the alert message.
Namespace: {{$labels.namespace}}/Pod: {{$labels.pod_name}}/Disk device: {{$labels.device}} usage exceeds 90%, current value {{ printf "%.2f" $value }}%
Alert notification
Simple Mode: You can set the Notification Objects, Notification Period, and Repeat Policy.
Standard Mode:
Do Not Specify Notification Policy: If you select this option, you can create a notification policy on the Notification Policy page after you create the alert rule. You can specify matching rules and conditions, such as the alert rule name, to match this rule. When an alert is triggered, the alert message is sent to the specified contact or contact group in the notification policy. For more information, see Notification policies.
Select a notification policy: If you select this option, ARMS adds a matching rule to the policy. This rule uses the alert rule ID to ensure the policy matches events from the current alert rule.
ImportantSpecifying a notification policy only ensures that alert events from this rule are matched by the selected policy. However, these events might also be matched by other notification policies that use fuzzy matching. The relationship between alert events and notification policies is many-to-many.
Do not specify notification policy
Advanced settings
Check interval
The interval at which the alert rule is evaluated. The default and minimum value is 1 minute.
1
Check when data is complete
Yes
No
Yes
Tag
Set tags for the alert. These tags can be used as matching conditions in notification policies.
N/A
Annotation
Set annotations for the alert.
N/A
After you configure the parameters, click Save. On the Prometheus page, you can view the status of the new alert rule.
If the Status of the alert rule is Automatic Interruption, edit the rule based on the reason provided, and then click Start. In the dialog box that appears, click Confirm. If you cannot resolve the issue, contact support by using the ARMS alert service DingTalk ID (d9j_rg9e4062f).
An alert rule might be automatically interrupted for the following reasons:
The query returns more than 1,500 time series.
No notification object is configured in the alert management settings.
The Prometheus instance is uninstalled or unavailable.
Manage alert rules
For rules created in the Managed Service for Prometheus console (both static threshold and custom PromQL types), you can perform the following actions: Edit, Start (for stopped rules), Stop (for running rules), Delete, Copy, and View alert event history.
For rules from other Alibaba Cloud services, you can view their alert event history and return to the source service's alert list.