Manage alerting rules as code using Managed Service for Prometheus rules

Best practices

Upload rules

1. Validate the rule file: Validate the rule file to check its integrity and PromQL expression syntax.

   promtool ruleup --dir /path/to/rules --check

   Sample output (validation successful):

   Found 1 rule file(s) in directory: /path/to/rules
   
   File: /path/to/rules/prometheus-rules.yml
   Type: prometheus
   Validation: OK

   Sample output (validation failed):

   Found 1 rule file(s) in directory: /path/to/rules
   
   File: /path/to/rules/prometheus-rules.yml
   Type: prometheus
   Validation errors:
     - group "system_resources": rule "HighCPUUsage": invalid PromQL expression: syntax error
     - group "application_services": rule "HighHTTPErrorRate": missing 'for' duration

   If the validation fails, the command exits with an exit code of 1.

2. Upload rules: After validation, upload the rules to Managed Service for Prometheus.

   promtool ruleup --dir /path/to/rules --upload --access-key-id YOUR_ACCESS_KEY_ID --access-key-secret YOUR_ACCESS_KEY_SECRET --user-id YOUR_USER_ID

   Interactive confirmation: The command displays the upload configuration and prompts for confirmation:

   === Upload Configuration ===
   User ID: 123456789
   Rule groups to upload: 5
     [1] system_resources (4 rules)
     [2] application_services (4 rules)
     [3] database (3 rules)
     [4] kubernetes (3 rules)
     [5] business_metrics (2 rules)
   Total rules to upload: 16
   Datasources: 2
     [1] Region: cn-hangzhou, Instance: rw-dfaa8e87409285b4b422b3c862ee
     [2] Region: cn-beijing, Instance: rw-abc123456789
   Workspace: will be automatically retrieved from instance
   
   Do you want to proceed with the upload? (yes/no): 

   Enter yes or y to continue. Any other input cancels the upload.

   Sample upload result:

   === Upload Result ===
   Total uploads: 10 (rule groups × datasources)
   Success: 10, Failed: 0
   
   Datasource [1]: Region=cn-hangzhou, Instance=rw-dfaa8e87409285b4b422b3c862ee
     Group: system_resources, Request ID: 12345678-1234-1234-1234-123456789abc
       Uploaded rules: 4
         - HighCPUUsage (ID: rule-001, Status: enabled, Level: warning)
         - HighMemoryUsage (ID: rule-002, Status: enabled, Level: warning)
         - HighDiskUsage (ID: rule-003, Status: enabled, Level: critical)
         - HighDiskIO (ID: rule-004, Status: enabled, Level: warning)
   
   Upload completed! 10 datasource(s) succeeded.

Delete rules from Managed Service for Prometheus

Delete specified rule groups from a Managed Service for Prometheus instance. The same validation performed for uploads also runs before deletion.

promtool ruleup \
  --dir /path/to/rules \
  --delete \
  --access-key-id YOUR_ACCESS_KEY_ID \
  --access-key-secret YOUR_ACCESS_KEY_SECRET \
  --user-id YOUR_USER_ID

Interactive confirmation: The command displays the deletion configuration and prompts for confirmation.

=== Delete Configuration ===
User ID: 123456789
Rule groups to delete: 5
  [1] system_resources
  [2] application_services
  [3] database
  [4] kubernetes
  [5] business_metrics
Datasources: 2
  [1] Region: cn-hangzhou, Instance: rw-dfaa8e87409285b4b422b3c862ee
  [2] Region: cn-beijing, Instance: rw-abc123456789

WARNING: This will delete all rules in the specified groups from the datasources!
Do you want to proceed with the deletion? (yes/no): 

Sample deletion result:

=== Delete Result ===
Total deletions: 10 (rule groups × datasources)
Success: 10, Failed: 0

Datasource [1]: Region=cn-hangzhou, Instance=rw-dfaa8e87409285b4b422b3c862ee
  Group: system_resources deleted, Request ID: 12345678-1234-1234-1234-123456789abc
  All rules in group deleted successfully

Deletion completed! 10 deletion(s) succeeded.

View basic information about rule files

Display basic information about all rule files in a directory, including the file type, rule group count, and data source count.

promtool ruleup --dir /path/to/rules

Sample output:

Found 2 rule file(s) in directory: /path/to/rules

File: /path/to/rules/prometheus-rules.yml
Type: prometheus
  PrometheusRule: 5 group(s), 2 datasource(s)
  Total rules: 15

File: /path/to/rules/alertmanager.yml
Type: alertmanager
  AlertManagerRule: 3 receiver(s)
  Route configured: yes

Complete workflow example

1. Validate rules before you upload them.

2. Use versioning:

   • Place rule files under a version control system such as Git.

   • Back up the existing configuration before you modify rules.

3. Manage by environment:

   • Use different rule file directories for different environments such as development, staging, and production.

   • Use different data source configurations to distinguish between environments.

4. Follow rule naming conventions:

   • Use meaningful names for rule groups, such as system_resources and application_services.

   • Use alert names that clearly describe the alert, such as HighCPUUsage and ServiceInstanceDown.

5. Configure data sources:

   • Ensure that the instance and regionId values in the data source configuration are accurate.

   • Verify that the instance ID exists and is accessible.

6. Confirm before batch operations:

   • Upload and delete operations can affect multiple data sources. Review the configuration carefully before you proceed.

   • Validate operations in a staging environment before you run them in production.

# 1. View basic information about the rule file
promtool ruleup --dir ./example/rule

# 2. Validate the rule file
promtool ruleup --dir ./example/rule --check

# 3. Upload the rules to Alibaba Cloud (interactive confirmation required)
promtool ruleup \
  --dir ./example/rule \
  --upload \
  --access-key-id $ALIYUN_ACCESS_KEY_ID \
  --access-key-secret $ALIYUN_ACCESS_KEY_SECRET \
  --user-id $ALIYUN_USER_ID

# 4. Delete the rules (interactive confirmation required)
promtool ruleup \
  --dir ./example/rule \
  --delete \
  --access-key-id $ALIYUN_ACCESS_KEY_ID \
  --access-key-secret $ALIYUN_ACCESS_KEY_SECRET \
  --user-id $ALIYUN_USER_ID

export ALIYUN_ACCESS_KEY_ID="your-access-key-id"
export ALIYUN_ACCESS_KEY_SECRET="your-access-key-secret"
export ALIYUN_USER_ID="your-user-id"

No rule files found in directory

• Cause: No rule files were found in the specified directory.

• Solution: Check whether the directory path is correct. Make sure the directory contains valid YAML rule files.

No PrometheusRule file found

• Cause: No files of the PrometheusRule type were found in the directory.

• Solution: Make sure at least one file contains a PrometheusRule configuration.

No rule groups found in PrometheusRule

• Cause: No rule groups are configured in the PrometheusRule file.

• Solution: Add at least one groups configuration to the rule file.

No datasources configured in PrometheusRule

• Cause: No data sources are configured in the rule file.

• Solution: Add a datasources configuration to the top of the rule file.

Datasource [N] missing instance field

• Cause: The instance field is missing or its value is empty in the data source configuration.

• Solution: Configure a valid instance value for each data source.

Datasource [N] missing regionId field

• Cause: The regionId field is missing or its value is empty in the data source configuration.

• Solution: Configure a valid regionId value for each data source.

Rule validation failed

• Cause: Rule validation failed. This can be caused by a PromQL syntax error or a missing required field.

• Solution: Use the --check parameter to view detailed validation error messages. Fix the errors and retry.

--access-key-id is required when --upload is set

• Cause: An AccessKey ID was not provided when --upload was used.

• Solution: Add the --access-key-id parameter.

--user-id is required when --upload is set

• Cause: A user ID was not provided when --upload was used.

• Solution: Add the --user-id parameter.