Overview and configuration of ARMS alert management-Application Real-Time Monitoring Service(ARMS)-阿里云帮助中心

This article describes best practices for unifying alerts from heterogeneous monitoring systems to help enterprises better address the challenges involved.

Background

In the cloud native era, as IT infrastructure grows in scale and complexity, organizations deploy more systems and services in cloud environments. To monitor these complex IT environments, organizations often use heterogeneous monitoring systems, such as Prometheus, Grafana, and Zabbix. This strategy helps them gather comprehensive monitoring data for deeper insights into the health and performance of their IT infrastructure.

However, using heterogeneous monitoring systems introduces challenges, most notably fragmented alerts. Each monitoring system generates its own alerts, scattering the information across different platforms. This fragmentation prevents organizations from getting a complete view of their IT systems' alert status. This makes responding to alerts more challenging and increases the complexity and workload of manual management.

To address these challenges, organizations need a unified and centralized alert management solution. Such a solution ensures that alerts reach the right personnel promptly, so they can take swift action to resolve potential issues.

Challenges of alert management

Use case 1: Inconsistent alerts from cloud services

In a typical cloud-native application deployment, the architecture includes products such as ACK, ECS, and RDS. Applications are deployed to ECS instances via Kubernetes and access data in RDS. This architecture typically uses the following monitoring products:

CloudMonitor monitors infrastructure resources, such as ECS and RDS, and triggers alerts when anomalies are detected.
Prometheus monitors Kubernetes and the pods deployed on it, and triggers alerts for Kubernetes-related issues.
ARMS monitors applications deployed on Kubernetes, including their traces, and triggers alerts when application errors occur.
SLS monitors application logs and triggers alerts for log anomalies.

Monitoring a system with multiple products forces users to repeatedly configure operational settings, such as contact configuration, notification methods, and on-call schedules, on each platform. Since alerts from these disparate systems are not integrated, it is difficult to quickly correlate them when an issue occurs.

Use case 2: Inconsistent alerts from heterogeneous systems

In multi-cloud or hybrid cloud environments, alerts from various monitoring systems become scattered and complex, creating significant operational challenges. Due to differences in data collection and processing methods across various cloud platforms and on-premises architectures, alerts generated by different systems often vary in format and behavior.

First, operations teams must spend significant time and effort triaging alerts scattered across various systems. Second, the lack of a unified view complicates alert management and analysis, slowing diagnosis and resolution. Furthermore, redundant or conflicting alerts from different systems add another layer of complexity to alert handling.

Use case 3: Integrating in-house and custom alerts

As systems scale, the "glue code" connecting their modules and systems inevitably grows. While this code is essential, it is often scattered, which makes it difficult to quickly detect and resolve issues. This difficulty threatens system high availability and stability. Consequently, integrating alerts from this code in a flexible and cost-effective way has become a major operational challenge.

Unified alert management

Building a unified alert management platform presents several challenges because different monitoring systems use unique alert definitions and processing workflows:

Inconsistent alert formats across systems result in high integration costs.
The variety of alert formats makes it difficult to standardize processing logic.
Alerting systems define alert severity in different ways.
Support for automatic recovery varies across alerting systems.

Features of ARMS Alert Management, such as integration, event processing flow, and notification policy, address the challenges of unified alert management.

Alert integration

Traditional alerts use a structured format suitable for a single source. However, aggregating data from multiple alert sources often causes schema conflicts. Therefore, ARMS uses semi-structured data to store alerts.

CloudMonitor alert data format:

Parameter	Type	Description
alertName	String	The name of the alert.
alertState	String	The state of the alert. Valid values: OK: Normal ALERT: Alerting INSUFFICIENT_DATA: No data
curValue	String	The value of the metric when the alert was triggered or resolved.
dimensions	String	The object that triggered the alert. Example: `{userId=110803419679**, instanceId=i-8psdh7l6lphbn10l**}`
expression	String	The alert condition.
groupId	String	The ID of the application group.
instanceName	String	The instance name.
lastTime	String	The duration of the alert in minutes.
metricName	String	The name of the metric. For more information, see `Metric Name` in the Cloud Service Metrics documentation.
metricProject	String	The name of the cloud service. This parameter is the same as `namespace`. For more information, see the Cloud Service Metrics documentation.
namespace	String	The namespace for the cloud service. This parameter is the same as `metricProject`. For more information, see the Cloud Service Metrics documentation.
preTriggerLevel	String	The severity level of the previously triggered alert. Valid values: CRITICAL: Critical WARN: Warning INFO: Informational
productGroupName	String	The name of the application group.
rawMetricName	String	The ID of the metric. For more information, see `Metric Id` in the documentation for Cloud Service Metrics.
regionId	String	The ID of the region.
regionName	String	The name of the region.
ruleId	String	The ID of the triggered alert rule.
timestamp	String	The time the alert was triggered.
transId	String	An ID that tracks a resource's transition from an alert state to a recovered state for a specific rule.
triggerLevel	String	The severity level of the currently triggered alert. Valid values: CRITICAL: Critical WARN: Warning INFO: Informational
unit	String	The unit of the metric. For more information, see `Unit` in the Cloud Service Metrics documentation.
userId	String	The user ID.

Zabbix alert data format:

Parameter	Type	Description
alertName	String	The name of the alert.
alertState	String	The state of the alert. Valid values: OK: Normal ALERT: Alerting INSUFFICIENT_DATA: No data
curValue	String	The value of the metric when the alert was triggered or resolved.
dimensions	String	The object that triggered the alert. Example: `{userId=110803419679**, instanceId=i-8psdh7l6lphbn10l**}`
expression	String	The alert condition.
groupId	String	The ID of the application group.
instanceName	String	The instance name.
lastTime	String	The duration of the alert in minutes.
metricName	String	The name of the metric. For more information, see `Metric Name` in the Cloud Service Metrics documentation.
metricProject	String	The name of the cloud service. This parameter is the same as `namespace`. For more information, see the Cloud Service Metrics documentation.
namespace	String	The namespace for the cloud service. This parameter is the same as `metricProject`. For more information, see the Cloud Service Metrics documentation.
preTriggerLevel	String	The severity level of the previously triggered alert. Valid values: CRITICAL: Critical WARN: Warning INFO: Informational
productGroupName	String	The name of the application group.
rawMetricName	String	The ID of the metric. For more information, see `Metric Id` in the documentation for Cloud Service Metrics.
regionId	String	The ID of the region.
regionName	String	The name of the region.
ruleId	String	The ID of the triggered alert rule.
timestamp	String	The time the alert was triggered.
transId	String	An ID that tracks a resource's transition from an alert state to a recovered state for a specific rule.
triggerLevel	String	The severity level of the currently triggered alert. Valid values: CRITICAL: Critical WARN: Warning INFO: Informational
unit	String	The unit of the metric. For more information, see `Unit` in the Cloud Service Metrics documentation.
userId	String	The user ID.

Nagios alert data format: The data fields for Nagios alerts include: host_name (host name), service_description (service description), state (the alert state: 0 for OK, 1 for WARNING, 2 for CRITICAL, and 3 for UNKNOWN), output (plugin output), long_output (long plugin output), perf_data (performance data), current_attempt (current check attempt), max_attempts (maximum check attempts), host_address (host address), notification_type (notification type), and timestamp (timestamp).

Semi-structured alert data structure

[ 
  {
   "labels": {
    "alertname": "<requiredAlertNames>",
    "<labelnames>": "<labelvalues>",
    ...
   }, 
   "annotations": {
   "<labelnames>": "<labelvalues>",
   }, 
   "startsAt": "<rfc3339>",
   "endsAt": "<rfc3339>",
   "generatorURL": "<generator_url>"
  },
  ...
]

labels: Alert metadata. A set of labels uniquely identifies an event. Events with an identical set of labels are considered a single event, and subsequent reports are merged. For example: alertname: alert name.
annotations: Annotations provide additional, non-identifying information for an alert event. Unlike labels, annotations are not considered alert metadata. For example: message: alert message. For a single recurring event, the labels remain the same, but the annotations can change. For example, the alert message might be: "CPU utilization for instance i-12b3ac3*** has exceeded 75% for three minutes. Current value: 82%."
startsAt: The time an alert event started.
endsAt: The time an alert event ended.
generatorURL: A backlink to the source of the alert event.

As shown in the code above, ARMS uses a semi-structured data structure based on the open-source Prometheus alert definition to describe alerts. This approach uses highly extensible key-value pairs, offering the flexibility to extend alert content and integrate alerts from different data sources.

Ingest custom alerts in any JSON format

ARMS can ingest alerts from any source in JSON format using a custom integration. To set up a custom integration, you first upload the JSON alert payload to the ARMS Alert Center. The configuration page has an upload area at the top and a field mapping configuration table at the bottom. Use this table to map fields from the original alert to ARMS standard fields, such as alertname, severity, and message.

ARMS defines key fields, such as alertname. You can configure additional extended fields in an integration to use in the alert processing logic. To map the hostname and hostip fields from an original alert message, navigate to the extended field configuration area. Click Add Extended Field to add the fields and configure their mappings.

Easy alert integration with common monitoring tools

ARMS offers built-in integrations to receive alerts from various on-premises and cloud-based monitoring systems. For a quick setup, refer to the integration overview. Supported systems include CloudMonitor, Prometheus, Grafana, Zabbix, Nagios, PagerDuty, and Datadog.

Standardized alert severity

ARMS classifies alerts into four levels: P1, P2, P3, and P4. You can use a mapping table to standardize alert levels from different systems. For example, you can map levels described as "L1," "Critical," or "Severe" to the P1 level. Each row in the alert level mapping table defines a mapping rule from a source level to a target level.

Unified handling of alerts with different formats

Since ARMS alerts use a semi-structured data model, you can use tags to standardize alert handling logic. You typically need at least two tags for this purpose. One tag, such as service or biz, defines the recipients. Another tag determines the notification channel and escalation policy. For example, the severity tag is commonly used to define the Service Level Agreement (SLA) for alert handling.

Severity	Definition	Acknowledgement time	Resolution time	Notification channel
P4	Minor issues that require action but do not affect customers.	24 hours	7 days	IM notification (such as DingTalk)
P3	Stability issues that require immediate attention from the on-call team, or minor issues affecting customers.	1 hour	24 hours	SMS and IM notifications (such as DingTalk)
P2	Critical system issues that severely impact many customers.	10 minutes	1 hour	SMS and IM notifications (such as DingTalk), with an escalation policy.
P1	Critical issues that require immediate contact with the management team.	1 minute	10 minutes	Phone call, SMS, and IM notifications (such as DingTalk), with an escalation policy.

ARMS provides notification and escalation policies to handle alerts of different severities. For configuration instructions, see Best practices for notification policies.

Tag design principles

When you design business tags for alert handling, follow these principles:

Mutually exclusive principle: Avoid assigning multiple tag keys for the same purpose to a single resource. For example, if you use the tag key service to identify a business, do not use other similar tag keys such as biz or business.
Collectively exhaustive principle: Assign the required tag keys and their corresponding tag values to all resources. For example, if an organization has three business units and uses the tag key service, it must have at least three distinct tag values, one for each business unit.
Limited values principle: Restrict a tag key's possible values to a finite list of core values, and avoid creating redundant or unnecessary values to simplify management. For example, if an organization has five business units, the tag key for business units must have only those five specific values.

In addition to business tags, you can define other tags to manage alerts, such as environment tags to separate alerts from your development and test environments. Adhering to these design principles simplifies your alert management configuration.

Alert enrichment

After you design tags, you need to apply them to alerts from various sources. In ARMS Alert Management, the low-code Event Flow lets you enrich alerts by adding tags with a visual, drag-and-drop configuration.

Scenario 1: Tagging alerts that match specific conditions
Consider a business that uses an in-house monitoring system. After integrating alerts from this system into ARMS Alert Management using a custom integration, you need to add the same business tag, for example, xx, to all of them. Configure an Event Flow as follows:
1. Log on to the ARMS console. In the left-side navigation pane, choose Alert Management > Event Flow, and then click Create Processing Flow.
2. In the panel that appears, create an Event Flow. Configure the trigger condition to match your custom integration. In the Trigger Condition section, select Integration Name as the match field, set the operator to Equals, and enter the name of the target integration, for example, xx-in-house-monitoring-system.
3. Add a Set Tag action. Set the tag name to service and the tag value to the corresponding business name.
Scenario 2: Extracting a tag by splitting a string

In an in-house alerting system, all hosts follow a fixed naming convention: ${env}-${biz}-${app}-${group}-${index}. You need to extract the biz field and use it as the business tag.

After you configure the appropriate trigger condition, add a Split Content action. In the action settings, set **Source Field** to hostname and **Delimiter** to -. Then, map the split values in sequence to the corresponding extended fields, such as env, service, app, and group.
Scenario 3: Enriching alerts with data from an Excel file

An application monitoring platform sends alerts that contain only an application ID. You need to enrich these alerts with additional information, such as the application name and owner, which are stored in an Excel file. The alert data contains only the appId field, while the Excel file (app_cmdb.xlsx) contains columns such as appId, appName, owner, and ownerPhone. Use appId as the lookup key.
1. Create an Excel data source and upload the app_cmdb.xlsx file. On the Data Source Management page, set **Data Source Type** to Excel. You can preview the data after the file is uploaded.
2. Configure an Event Flow and add a Field Enrichment action. Select the data source that you created in the previous step. In the action settings, set the Match Field to appId and configure the output field mappings to populate the appName, owner, and ownerPhone extended fields with data from the Excel file.
Scenario 4: Enriching alerts by calling an external service with Serverless (Function Compute)

Similar to Scenario 3, when alert data is missing and must be fetched from an external system such as a CMDB, you can use an API-type data source for alert enrichment.
1. Create a Function Compute application and develop an HTTP service that accepts appId as an input parameter and returns parameters such as appName, owner, and ownerPhone. For sample code, see the Additional information section below.
2. Create an API data source. On the Data Source Management page, set **Data Source Type** to API and **Request Method** to POST. For the URL, enter the HTTP service URL of the Function Compute application from the previous step.
3. Configure an Event Flow and add a Field Enrichment action. Select the API data source. Set the Match Field to appId and map the fields from the API response to the appName, owner, and ownerPhone extended fields.

Configure automatic alarm recovery

Monitoring systems handle alert auto-recovery differently. For example, Prometheus does not send a dedicated recovery alert; instead, it uses an alert's end time to mark its resolution. In contrast, CloudMonitor integrates the recovery status into its alert levels.

Parameter: triggerLevel
Data type: String
The alert level. Valid values are:
- CRITICAL: Critical
- WARN: Warning
- INFO: Information
- OK: Normal

The logic for recovering an alert varies by scenario. For threshold-based alerts, the alert recovers when the monitored value no longer meets the threshold condition. In contrast, critical, event-based alerts occur instantaneously and lack a natural recovery process. An operator must manually recover these alerts after confirming the event's impact has been resolved.

Scenario 1: Configure an auto recovery time for non-recovering alerts

Event-based alerts often require manual investigation to assess their impact before recovering them. Automatically recovering them too quickly can lead to unaddressed issues. For these cases, you can disable auto-recovery or set a long duration to give your team enough time to investigate the alert and its impact.

For custom ARMS integrations, you can configure the auto recovery time. The Auto Recovery Time drop-down list includes the following options: Do Not Recover, 5 Minutes, 10 Minutes, 30 Minutes, 1 hour, 3 hours, 6 hours, 12 hours, and 24 hours. Select a time appropriate for the alert type.
Scenario 2: Configure an alert recovery field to recover an alert upon receiving a recovery event

In ARMS alert integrations, you can configure an alert recovery field. When the value of a field in an incoming alert payload meets a specified condition, ARMS treats it as a recovery event. It then uses other fields in the payload to find and recover the corresponding active alert. The following diagram illustrates this process.

To configure this in the ARMS console, go to the custom integration configuration page. In the Alert Recovery section, enable the feature. Then, configure the Alert Recovery Field (for example, status) and set the recovery condition (for example, the alert recovers when the field's value is ok or resolved). You can also configure Deduplication Fields to ensure the recovery event matches the original alert.

To resolve an alert, one of the following conditions must be met:

If no deduplication field is defined, the original alert and the alert resolution event must have the same tags.
If a deduplication field is defined, that field must have the same value in both the original alert and the alert resolution event.

Note

When you configure a field, such as status, as an alert resolution field, do not add it to the mapping rules. This typically causes a field mismatch between the original alert and the alert resolution event, which prevents the alert from resolving.

Additional information

Sample Function Compute code:

# -*- coding: utf-8 -*-

import logging
import json

def handler(environ, start_response):
    context = environ['fc.context']
    request_uri = environ['fc.request_uri']
    body_str = get_request_body(environ)
    id = json.loads(body_str).get('appId')
    # This pseudocode line queries a CMDB for application details.
    # The returned app object uses the following format:
    # {"appId":"b38cdf95-2526-4d7a-9ea9-ffe7b32*****", "appName": "iot-iam", "owner":"John Doe", "ownerPhone": "130xxxx1236"}
    app = cmdb.getApp(id)
    ret = json.dumps(app)
    status = '200 OK'
    response_headers = [('Content-type', 'text/plain')]
    start_response(status, response_headers)
    return [ret.encode('utf-8')]

def get_request_body(environ):
    try:
        request_body_size = int(environ.get('CONTENT_LENGTH', 0))
    except (ValueError):
        request_body_size = 0
    request_body = environ['wsgi.input'].read(request_body_size)
    return request_body