Create a Prometheus alert rule using a PrometheusRule-Managed Service for Prometheus(Prometheus)-阿里云帮助中心

Learn how to create an alert rule in Managed Service for Prometheus by using a PrometheusRule.

Background information

A PrometheusRule is a Custom Resource Definition (CRD) in Prometheus Operator used to manage alerts. A PrometheusRule defines a group of Prometheus alerting rules or recording rules. In a Kubernetes cluster with Prometheus Operator installed, you can use a PrometheusRule to create alerts. As an implementation of Monitoring as Code, PrometheusRule is ideal for building GitOps-based development and operations (DevOps) systems.

You can also use a PrometheusRule to create an alert rule in Managed Service for Prometheus. Managed Service for Prometheus provides a user-friendly UI for managing alerts and is also compatible with creating alerts by using a PrometheusRule in Prometheus Operator.

Usage notes

Managed Service for Prometheus supports the following two methods for creating and managing alert rules by using a PrometheusRule.

Method 1: Import a PrometheusRule as an alert rule template to visually create and update alerts in batches. This method is ideal for users who prefer to use a UI to efficiently manage numerous alert rules in the Alibaba Cloud console. For more information, see Method 1: Import a PrometheusRule as an alert rule template.
Method 2: Enable the agent of Managed Service for Prometheus to automatically synchronize PrometheusRules. This allows you to continue using your open source alert configurations. This method is suitable for users who are accustomed to open source practices and want to build a Monitoring as Code-based operations system. For more information, see Method 2: Enable the agent to synchronize PrometheusRule alert rules.

Note

By default, Managed Service for Prometheus does not read PrometheusRules created in Kubernetes. Therefore, you can use either method to create and manage alerts by using a PrometheusRule based on your business requirements.

Method 1: Import PrometheusRule as a template

Benefits

Managed Service for Prometheus provides the alert rule template feature to simplify the configuration of alerts across multiple regions and clusters. After you configure an alert rule template once, you can apply it to different clusters worldwide. This significantly reduces the configuration workload and improves efficiency.

Managed Service for Prometheus allows you to import PrometheusRule configurations as alert rule templates in batches, which avoids the need to create a separate PrometheusRule for each cluster.

Procedure

Log on to the Cloud Monitor console.
In the left-side navigation pane, click Alert Rule Template.
On the Prometheus Alert Rule Templates page, click Import Templates in the lower-left corner.
In the Import dialog box, paste the content of the spec field from your PrometheusRule into the text box, and then click Import Alert Rule Template.
Example: The following example shows the content of the spec field in a sample PrometheusRule:
```
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  creationTimestamp: null
  labels:
    prometheus: example
    role: alert-rules
  name: prometheus-example-rules
spec:
  groups:
  - name: ./example.rules
    rules:
    - alert: ExampleAlert
      expr: vector(1)
      for: 1m
      labels:
        severity: critical
      annotations:
        message: testalert
```
You need to paste the spec content into the Import dialog box and click Import Alert Rule Template.
After a successful import, select the new template and click Apply Templates in the lower-left corner of the Prometheus Alert Rule Templates page to apply it in batches.

Method 2: Enable the Managed Service for Prometheus agent to synchronize PrometheusRules

Use cases

You use systems like GitOps and want to continue configuring alert rules by using the Monitoring as Code approach.
You want to create alerts by using a PrometheusRule for an ACK cluster that is connected to Managed Service for Prometheus. This method does not apply to ACS clusters.

Procedure

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, find your cluster and click Applications in the Actions column.
On the Deployments page, select arms-prom from the Namespace drop-down list. In the Name column, find the Deployment whose name starts with arms-prometheus, such as arms-prometheus-ack-arms-prometheus. Then, click Edit in the Actions column.
On the Edit page, in the Lifecycle section, configure the Parameter for Start. Add the --alert=true argument to the Parameter field. Then, in the upper-right corner of the Edit page, click Update.

After the Managed Service for Prometheus agent is updated, wait for 3 to 5 minutes. You can then see that the status of all alerts is displayed as running on the Alert Rule List page in the Managed Service for Prometheus console.

Important

After you enable automatic synchronization of PrometheusRules for your cluster, do not modify the synchronized alert rules in the Managed Service for Prometheus console. This can cause unexpected behavior. For example, you cannot rename or delete alert rules in the console. Additionally, modifying these rules in the Managed Service for Prometheus console disrupts the Monitoring as Code mechanism. This means that the modified rules cannot be synchronized to external systems, such as GitOps.

What to do next

After you enable alert synchronization for the Managed Service for Prometheus agent, the Managed Service for Prometheus agent creates a set of built-in alert rules in the arms-prom Namespace, as shown in the following output. For more information about these automatically created built-in alert rules, see Create a Prometheus alert rule.

$kubectl get prometheusrule -n arms-prom
NAME                    AGE
prometheus-arms-rules   439d
prometheus-event-rules  439d
prometheus-k8s-rules    439d

If you do not want these built-in alert rules, you can disable their automatic creation by modifying the Managed Service for Prometheus agent's defaultAlert parameter.

Note

The defaultAlert parameter controls the automatic creation of default alerts. By default, this parameter is set to true. To prevent the creation of default alerts, set this parameter to false.

Set the defaultAlert parameter of the Managed Service for Prometheus agent to false.
1. Log on to the ACK console. In the left navigation pane, click Clusters.
2. On the Clusters page, find the target cluster and click Applications in the Actions column.
3. On the Deployments page, select arms-prom from the Namespace drop-down list. In the Name column, find the Deployment whose name starts with arms-prometheus. Then, click Edit in the Actions column.
4. On the Edit page, in the Lifecycle section, configure the parameter for Start. Add --defaultAlert=false to the Parameter text box. Then, in the upper-right corner of the Edit page, click Update.
Note
After the Managed Service for Prometheus agent is updated, wait for 3 to 5 minutes. Managed Service for Prometheus will no longer create default alerts for the cluster.
Optional: Delete the default alert rules.
1. Log on to the Managed Service for Prometheus console.
2. In the left navigation pane, click Instances.
3. Click the name of the Prometheus instance for the target Kubernetes cluster.
4. In the left-side navigation pane, click Alert Rules. On the Prometheus Alert Rules page, find the alert rule that you want to delete, click Delete in the Actions column, and then click OK in the dialog box that appears.