Enrich Prometheus alert rules with entity information-Cloud Monitor(CMS)-阿里云帮助中心

Native Prometheus alert labels usually carry only dimension strings such as instance, pod, namespace, and job. They indicate nothing beyond a metric series crossing a threshold and cannot be associated with an entity in UModel. Cloud Monitor 2.0 lets you declare entity labels as custom keys in the annotations of a Prometheus alert rule to bind the alert to a specific entity, such as an application, a pod, an operation, or a self-managed business object. This tutorial walks you through declaring entity labels and verifying that the entity association takes effect.

Business scenario and expected results

The downstream capabilities of Cloud Monitor 2.0 depend on a stable entity identity, which consists of three elements:

Entity domain (domain)
Entity type (entity_type)
Entity ID (entity_id)

After you complete the configuration in this topic, alert events include the _cms_domain, _cms_entity_type, and _cms_entity_id fields.

The following table compares the capabilities before and after entity enrichment.

Capability	Without entity enrichment	With entity enrichment
Alert aggregation	Alerts can be aggregated only by alert rule.	Repeated alerts for the same entity within the convergence window can be aggregated by entity.
Root cause analysis	Entity relationships cannot be drilled up or down.	Upstream and downstream drill-down is available along the entity relationship chain.
Silencing and suppression	Matching is available only by label combination.	Precise control is available for a single entity.
Event Center	Events lack a stable entity identity.	Events are attached to entity nodes and can be retrieved and governed by entity.

Prerequisites

Cloud Monitor 2.0 is activated, and a workspace is created.
A Prometheus data source is connected to the workspace, and a Prometheus alert rule is created or is to be created.
The target entity type exists in the Cloud Monitor UModel metadata table. Standard ACS cloud service, Application Performance Management (APM) application in Application Real-Time Monitoring Service (ARMS), and Container Service (k8s) types are preset by Cloud Monitor and require no registration. For self-managed business entities, register the entity_type and its primaryKeys first.

Step 1: Confirm the entity domain and entity type

Identify the alert object of the alert rule, that is, the type of entity to which the alert must finally be attached.

Determine the values of custom_cms_domain and custom_cms_entity_type based on the following table.

Entity domain (domain)	Applicable object	entity_type example
apm	Applications and operations integrated with Application Monitoring in ARMS	`apm.service`, `apm.operation`
acs	Alibaba Cloud service instances	`acs.vpc.peerconnection`
k8s	Workloads in Container Service clusters	`k8s.pod`
rum	Frontend user experience monitored objects	`rum.service`
Custom	Self-managed business entities	`mybiz.channel`

Confirm that the series of the alert metric contains the label that corresponds to the primaryKey of the entity type. For example, the primaryKey of apm.service is service_id, and the corresponding label is acs_arms_service_id. If the metric does not contain this label, the backend cannot automatically generate the entity_id.

Note

An alert rule can declare only one entity dimension. If the same metric requires alerts by both service and operation, split it into two rules instead of declaring multiple entity_type values in one rule.

Step 2: Add entity labels to the alert rule

Entity labels are custom keys in the annotations section of a Prometheus alert rule.

Open the Prometheus alert rule for editing in the Cloud Monitor 2.0 console.
Declare the following custom keys in the annotations of the alert rule. After the keys are declared, the backend automatically completes the remaining enrichment work by using SPL.
Save and publish the alert rule.

The following table describes the parameters.

Parameter	Required	Description
`custom_cms_domain`	Yes	The domain to which the entity belongs. Valid values include apm, acs, k8s, rum, and custom domain names.
`custom_cms_entity_type`	Yes	The specific entity type, such as `apm.service`, `acs.vpc.peerconnection`, or `rum.service`. Go Template syntax is supported.
`custom_cms_entity_id`	No	Specifies the entity ID directly. If you leave this parameter blank, the backend queries the preset metadata table based on the domain and entity_type, and automatically generates the entity ID by using primaryKeys and fieldMapping.
`custom_cms_entity_id_fields`	No	Provides key-value pairs in the `a=b,c=d` format when the entity ID requires additional variable assignments. This parameter is generally not required.

Example 1: Standard cloud services and APM applications

When the alert metric belongs to an Alibaba Cloud service (ACS) or an APM service, declare only the two required labels. The backend calculates the entity_id automatically based on the metadata table.

The following example shows an APM application:

annotations:
  custom_cms_domain: apm
  custom_cms_entity_type: apm.service

The following example shows a VPC peering connection:

annotations:
  custom_cms_domain: acs
  custom_cms_entity_type: acs.vpc.peerconnection

The backend generates the entity_id by retrieving the value from the instanceId label. Because the corresponding field mapping rule (instance_id to instanceId) is built-in, you do not need to declare it in an alert rule.

Example 2: Alerts by operation or RPC dimension

To split alerts by operation within the same APM application, refine the entity to "service + operation". Combine two labels with a Go Template:

annotations:
  custom_cms_domain: apm
  custom_cms_entity_type: "{{ if $labels.rpc }}apm.operation{{ else }}apm.service{{ end }}"
  custom_cms_entity_id: "{{ if $labels.rpc }}{{ printf \"%s-%s\" $labels.acs_arms_service_id $labels.rpc | md5 }}{{ else }}{{ printf \"%s\" $labels.acs_arms_service_id | md5 }}{{ end }}"

Example 3: Self-managed or custom business entities

For a self-managed business entity, declare the custom domain and entity type, and construct the entity ID from your business labels with a Go Template:

annotations:
  custom_cms_domain: mybiz
  custom_cms_entity_type: mybiz.channel
  custom_cms_entity_id: "{{ printf \"%s-%s\" $labels.tenant_id $labels.channel_id | md5 }}"

Step 3: Verify that the entity association takes effect

After you publish the rule, confirm that entity enrichment takes effect in any of the following ways.

View the event details in Event Center: Log on to the Cloud Monitor 2.0 console, select the target workspace, and go to Event Center. Open the details of the event triggered by the rule and confirm that domain, entityType, and entityId under resource.entity are all non-empty.
Preview on the alert rule details page: Trigger a preview query on the alert rule details page. If the returned series carries the _cms_entity_id field, which is a 32-character hexadecimal string, SPL enrichment takes effect.
Check the result in the Entity Topology view: In the Entity Topology view of Event Center, the alerting entity is highlighted, and the color changes with the alert level.