Create alert rules using PrometheusRule

Use PrometheusRule to create alert rules as code in Managed Service for Prometheus.

Background

PrometheusRule is a Custom Resource Definition (CRD) from Prometheus Operator for alerting. A PrometheusRule defines Prometheus alerting and recording rules. In Kubernetes clusters with Prometheus Operator, create alerts by deploying PrometheusRules. This Monitoring as Code approach integrates with GitOps-based DevOps workflows.

Managed Service for Prometheus supports PrometheusRule-based alerting alongside its built-in visual alert management in the Managed Service for Prometheus console.

Usage notes

Managed Service for Prometheus supports two methods to create alert rules with PrometheusRule.

Import PrometheusRules as alert rule templates to batch-create and update alerts from the console. Method 1: Import PrometheusRules into alert rule templates.
Enable PrometheusRule synchronization in the Managed Service for Prometheus plug-in for Monitoring as Code alert configuration. Best for GitOps workflows. Method 2: Enable the plug-in to synchronize PrometheusRule alert rules.

Note

By default, Managed Service for Prometheus does not read PrometheusRules created in Kubernetes. Choose either method based on your requirements.

Method 1: Import PrometheusRules into alert rule templates

Benefits

The Create Prometheus alert rule templates feature in Managed Service for Prometheus lets you configure an alert rule template once and apply it across regions and clusters.

With Managed Service for Prometheus, import PrometheusRules as alert templates in batches instead of creating rules for each cluster individually.

Procedure

Log on to the Cloud Monitor console.
In the navigation pane on the left, click Alert Rule Templates.
In the lower-left corner of the Prometheus Alert Rule Templates page, click Batch Import Templates.
In the Import dialog box, paste the spec content of the PrometheusRule into the text box, and then click Import Alert Rule Template.

Example: The area marked with ① in the following figure shows the spec content of a sample PrometheusRule:

Paste the spec content (①) into the Import dialog box and click Import Alert Rule Template, as shown in the following figure:

After the import succeeds, select the imported template. In the lower-left corner of the Prometheus Alert Rule Templates page, click Batch Apply Templates to apply it to your Prometheus alert rules.

Method 2: Enable the Managed Service for Prometheus plug-in to synchronize PrometheusRule alert rules

Scenarios

You use GitOps or similar systems to configure alerts with a Monitoring as Code approach.
You want to create PrometheusRule-based alerts for ACK clusters connected to Managed Service for Prometheus. Not applicable to ACS.

Procedure

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, find the destination cluster and click Application Management in the Actions column.
On the Stateless page, set the namespace to arms-prom. In the Name column, find the deployment whose name starts with arms-prometheus, such as arms-prometheus-ack-arms-prometheus. Then, click Edit in the Actions column.
On the Edit page, in the Lifecycle area, configure the Parameters for Start Execution by adding --alert=true to the Parameters text box, and then in the upper-right corner of the Edit page, click Update.

After the Managed Service for Prometheus plug-in is updated, wait for 3 to 5 minutes. On the Alert Rule List page in the Managed Service for Prometheus console, the status of all alerts changes to Enabled.

Important

After PrometheusRule synchronization is enabled for a cluster, do not modify the synchronized alert rules in the Managed Service for Prometheus console. Console modifications cause unexpected behavior. For example, you cannot perform operations such as modifying rule names or deleting rules in the console. In addition, modifying these rules in the Managed Service for Prometheus console disrupts the Monitoring as Code mechanism, preventing the rules from synchronizing with external O&M systems such as GitOps.

What to do next

After you enable the alert feature for the Managed Service for Prometheus plug-in, the Managed Service for Prometheus plug-in creates built-in alert rules in the arms-prom namespace. Create a Prometheus alert rule .

To disable built-in alert rules of the Managed Service for Prometheus plug-in, modify the defaultAlert parameter:

Note

The defaultAlert parameter controls automatic creation of default alerts. Set it to true to enable or false to disable.

Set the defaultAlert parameter of the Managed Service for Prometheus plug-in to false.
1. Log on to the ACK console. In the left navigation pane, click Clusters.
2. On the Clusters page, find the destination cluster and click Application Management in the Actions column.
3. On the Stateless page, set the namespace to arms-prom. In the Name column, find the deployment whose name starts with arms-prom. Then, click Edit in the Actions column.
4. In the Lifecycle area on the Edit page, add --defaultAlert=false to the Parameters text box under Start Execution, and then click Update in the upper-right corner of the Edit page.
Note
After the Managed Service for Prometheus plug-in is updated, wait for 3 to 5 minutes. Managed Service for Prometheus no longer creates default alerts for the cluster.
Optional:Delete the default alert rules.
1. Log on to the Managed Service for Prometheus console.
2. In the left navigation pane, click Instances.
3. Click the name of the Prometheus instance for the destination Kubernetes cluster.
4. In the navigation pane on the left, click Alert Rules. On the Prometheus Alert Rules page, find the alert rule that you want to delete and click Delete in the Actions column. In the dialog box that appears, click OK.