DocsICP FilingConsole
官方文档
首页

Unified alerting with ARMS Alert Management

更新时间：
复制 MD 格式
我的收藏

Overview

In the cloud-native era, enterprise IT infrastructures are growing larger. More systems and services are deployed in cloud environments. To monitor these complex IT environments, companies often use heterogeneous monitoring systems, such as Prometheus, Grafana, and Zabbix. This helps them obtain more comprehensive monitoring data to better understand the health and performance of their IT infrastructure.

However, these heterogeneous monitoring systems also create problems, the most significant of which is scattered alert information. Different monitoring systems generate different alerts, which can be spread across various systems. This makes it difficult for companies to obtain a complete view of their IT system's alert status. This makes responding to alerts more difficult and increases the complexity and workload of manual management.

This document describes a unified and centralized alert management solution based on Application Real-Time Monitoring Service (ARMS). This solution ensures that alerts reach the right people promptly so they can take the necessary actions to address potential issues quickly.

Advantages

Extensive support for alert sources

ARMS Alert Management covers more than 10 common alert sources, including CloudMonitor, Simple Log Service (SLS), Prometheus, Grafana, Zabbix, Open-Falcon, and Nagios. It also provides an API for integration. This allows users to process alerts from any source in a unified way. Companies can manage and respond to alerts from different systems on a single platform. This simplifies O&M processes.

Integration with multiple collaboration tools

By integrating with team collaboration tools such as DingTalk, WeCom, Lark, Apsara DevOps, Jira, and PagerDuty, ARMS Alert Management lets you handle alert events directly within these tools. You do not need to switch to other platforms or systems. This not only improves work efficiency but also reduces security risks from frequent account switching. It also supports reusing existing account systems and provides flexible permission allocation, further enhancing security and convenience.

Customer scenarios

Inconsistent alerts from cloud products after migrating to the cloud

When a company migrates its business to the cloud, it might use multiple Alibaba Cloud products, such as Container Service for Kubernetes (ACK), Elastic Compute Service (ECS), and ApsaraDB RDS, to build its application architecture. Each product has its own monitoring system, which leads to alert information being scattered across different platforms. ARMS Alert Management can consolidate these alerts from different systems, provide a unified view, simplify the configuration process, and support a more efficient alert handling workflow.

Inconsistent alerts from heterogeneous monitoring systems in multicloud or hybrid cloud architectures

For companies that use a multicloud strategy or maintain both private and public cloud resources, the challenge is to effectively manage and respond to alert signals across environments. ARMS Alert Management supports the manual integration of more types of monitoring tools. This helps companies achieve integrated alert management across cloud platforms, which improves O&M efficiency and reduces potential risks.

Need to integrate self-developed monitoring systems and custom event alerts

As application complexity increases, development teams may also create custom monitoring logic for specific needs. The ARMS alert management system allows users to flexibly integrate alerts from these custom monitoring sources. This ensures all key metrics are monitored in real time and appropriate actions are taken.

Solution architecture

This solution is based on ARMS Alert Management. It integrates alert sources such as ARMS, Simple Log Service (SLS), CloudMonitor, and self-managed Prometheus. It uses an event processing flow to compress, reduce noise in, and deduplicate collected alerts. It also configures notification recipients and notification policies. This provides end-to-end management of alert information, from collection and processing to precise notification. This helps improve O&M efficiency and the accuracy of alert handling.

Product fees and terms

Product fees

Product Name

Description

Fee

ARMS Alert Management

ARMS Alert Management provides reliable alert convergence, notifications, and automatic upgrades to help you quickly detect and resolve business alerts. This topic describes the architecture and benefits of alert management.

Some notification channels are subject to fees. For more information, see Product Billing.

ARMS Prometheus Monitoring

ARMS Prometheus Monitoring is fully compatible with the open-source Prometheus ecosystem, supports monitoring for a wide range of components, provides multiple out-of-the-box dashboards, and offers a fully managed Prometheus service. With ARMS Prometheus Monitoring, you do not need to build your own Prometheus monitoring system and therefore do not have to worry about underlying data storage, data visualization, or system O&M.

Paid. For more information, see Product Billing.

CloudMonitor

CloudMonitor is a service for monitoring Alibaba Cloud resources and internet applications. You can use CloudMonitor to collect metrics from Alibaba Cloud resources, detect the availability of internet services, and set alerts for metrics.

Some features are free, while advanced features are paid. For more information, see Product Billing.

Simple Log Service

Simple Log Service (SLS) is a cloud-native observability platform. It provides a large-scale, low-cost, and real-time platform service for data such as logs, metrics, and traces. SLS offers one-stop services for data ingestion, processing, query and analysis, visualization, alerting, consumption, and delivery. It enhances your digital capabilities in scenarios such as R&D, O&M, operations, and security.

Paid. For more information, see Product Billing.

Glossary

Name

Description

Period-over-period increase/decrease %

If β is the data from the last N minutes (can be average, sum, max, or min), and α is the data from 2N minutes ago to N minutes ago, the period-over-period comparison is between β and α.

YoY Change (%)

For an hour-over-hour comparison, if β is the data from the last N minutes (can be average, sum, max, or min), and α is the data from the last N minutes of the previous hour, the hour-over-hour comparison is between β and α.

Notification policy

A notification policy determines how events are aggregated into a single alert and how assigned alerts are sent to contacts. You can configure notifications through phone calls, text messages, emails, or DingTalk groups to inform contacts about alerts that need to be resolved or acknowledged within one minute.

Escalation policy

If an alert escalation times out, Alert Management notifies different batches of contacts defined in the escalation rule in sequence. Escalation policies can be added to specific notification policies.

Security

Security settings

To ensure security when you use custom robots, add a custom keyword in the robot's security settings. For example, if you add the custom keyword alert, messages sent by this robot must contain the word alert to be sent successfully.

Access control

Resource Access Management (RAM) lets you use RAM users to segregate permissions. You can grant permissions to RAM users as needed and avoid the security risks that are associated with exposing your Alibaba Cloud account's AccessKey. For more information about how to grant permissions to RAM users using custom authorization policies, see Custom RAM authorization policies for Alert Management.

Notes

Free alert notification quota limit

Accounts that have not activated the ARMS Alert Management service receive 15 free text message notifications and 3 free phone call notifications per day. Exceeding the free quota will cause alert notifications to fail. If you need more alert notifications, activate the Alert Management service.

Implementation steps

Preparations

  • Ensure that you have configured alerts in the alert sources to be integrated, such as CloudMonitor, ARMS, and Simple Log Service.

  • To create a DingTalk, WeCom, or Lark alert robot, ensure that you have created an alert group in the corresponding client.

Implementation duration

After completing the preparations, the estimated implementation time for this solution is 60 minutes.

Procedure

Integrate alert sources

Integrate ARMS alerts

ARMS Alert Management has integrated ARMS by default. You can search for ARMS-DEFAULT in Alert Management > Integration to view the ARMS integration.

Integrate Simple Log Service alerts

ARMS Alert Management supports integrating alerts from Simple Log Service through a webhook. This lets you manage these alerts in a centralized way.

There are two ways to integrate Simple Log Service with ARMS Alert Management:

Create an integration and connect to Simple Log Service

  1. Log on to the ARMS console. In the navigation pane on the left, choose Alert Management > Integration.

  2. On the Integration page, on the Alert Integration tab, click Simple Log Service.

  3. In the dialog box that appears, enter a name and description for the Simple Log Service integration, and set the automatic alert event recovery time.

  4. In the All Projects area of Projects to Integrate Alert Events, all projects of Alibaba Cloud Simple Log Service are displayed. Select the project whose alerts you want to integrate, click the icon, and then click Save.

    This feature automatically traverses all alert rules configured in the selected project. It adds the alert request address https://alerts.aliyuncs.com/api/v1/events/{{token}} to these rules, where {{token}} is the key automatically generated when the integration is created.

After the configuration is complete, you can view the created Simple Log Service integration on the Alert Integration tab.

Connect to a specified alert rule in Simple Log Service

  1. On the Alert Integration tab of the Integration page, copy the integration URL of the target Simple Log Service integration.

  2. Log on to the Simple Log Service console.

  3. In the Project List area, click the target project. In the navigation pane on the left, click the icon, which is the alert icon.

  4. On the Alert Center page, choose Notification Recipient > Webhook Integration. On the Webhook Integration page, click Create.

  5. In the dialog box that appears, add the universal webhook for ARMS Alert Management. Set the name to ARMS Alert Management, select Universal Webhook for the type, and configure other parameters as prompted.

  6. On the Alert Center page, choose Notification Policy > Content Template and click Create. In the dialog box that appears, click the WebHook-Custom tab. Set Send Method to Send one by one and configure the content to send as shown in the following figure. Then, click OK.

    {
  "uid": "{{ alert.aliuid }}",
  "project": "{{ alert.project }}(https://sls.console.aliyun.com/#/project/{{ alert.project }}/categoryList)",
  "trigger": "{{ alert.alert_name }}",
  "condition": "{{ alert.condition }}",
  "context": {{ alert.results[0].raw_results | to_json | quote }},
  "message": " [Uid] {{ alert.aliuid }}\n\n> [Project] [{{ alert.project }}](https://sls.console.aliyun.com/#/project/{{alert.project }}/categoryList)\n\n> [Trigger] {{ alert.alert_name }}\n\n> [Condition] {{ alert.alert_name }}\n\n> [Message] Notification Content\n\n> [Context] {{ alert.results[0].raw_results | to_json | quote | removeprefix('"') | removesuffix('"') }}\n\n> [View Details]({{ alert.query_url }})"
}

  7. On the Alert Center page, click the Alert Rules tab. In the Operation column of the target rule, click Edit. In the dialog box that appears, expand Advanced Configuration. In the Output Target area, select SLS Notification. In the Alert Policy area, select Simple Mode. Configure the notification channel for the universal webhook as prompted. After the configuration is complete, click OK.

Integrate CloudMonitor alerts

ARMS Alert Management supports integrating alerts from CloudMonitor through a webhook. After alerts are integrated, you can manage them centrally. This solves the problem of scattered alert configurations and inconsistent alert handling capabilities across different systems.

There are two ways to integrate CloudMonitor with ARMS Alert Management:

Create an integration and connect to CloudMonitor

  1. Log on to the ARMS console. In the navigation pane on the left, choose Alert Management > Integration.

  2. On the Alert Integration tab, click CloudMonitor.

  3. In the Create CloudMonitor Integration dialog box, enter a name and description for the CloudMonitor integration, and set the automatic alert event recovery time.

  4. In the All Products area of Products to Integrate Alert Events, the products supported by CloudMonitor for alert integration are displayed. Select the products whose alerts you want to integrate, click the icon, and then click Save.

After the configuration is complete, you can view the created CloudMonitor integration on the Alert Integration tab.

Connect to a specified alert rule in CloudMonitor

Before you can connect to a specified CloudMonitor alert rule, you must first create a CloudMonitor integration. For more information, see Create an integration and connect to CloudMonitor.

  1. On the Alert Integration tab of the Integration page, copy the integration URL of the target CloudMonitor integration.

  2. Log on to the CloudMonitor console.

  3. In the navigation pane on the left, choose Alert Service > Alert Rules.

  4. On the Alert Rules page, in the Operation column of the target alert rule, click Modify.

  5. In the panel that appears, in the Advanced Settings area, set the alert callback URL to the integration URL of the CloudMonitor integration.

  6. Click OK.

Edit an integration

ARMS Alert Management provides preset field mappings between the Simple Log Service alert source and ARMS Alert Management. You can also add or modify field mappings on the Edit Integration page as needed.

  1. On the Alert Integration tab, in the Operation column of the target integration, click Edit.

  2. In the Event Mapping area on the left, click Send Test Data.

  3. In the Send Test Data dialog box, enter the alert content from the third-party alert source in JSON format, and then click Send.

    • If the message Upload successful, but no event was generated. Please configure event mapping rules based on the raw data! is displayed, the fields of the alert source are not mapped to the ARMS alert event fields. The data that is sent is saved in the record box on the left. This lets you directly select the corresponding alert source fields when you configure the mapping.

    • If Upload successful! is displayed, the corresponding alert content is reported to the Alert Event History page. For more information, see View Alert Event History.

  4. In the Send Test Data dialog box, click Close.

  5. In the Event Mapping area on the left, click to expand the alert data for which you want to create a mapping.

  6. In the Event Mapping area on the right, configure the mapping between the alert source fields and the ARMS alert fields.

    1. Optional: In the Select Root Node area, choose whether to use batch processing. When the alert data contains an array node, you can specify the target array node as the root node to batch process the data under that root node. After selecting Use Batch Processing, select the array node to be batch processed as the root node.

    When the alert data contains multiple array nodes, ARMS Alert Management supports selecting only one of them for batch processing.

    1. Optional: Select Configure Alert Recovery Event and set the recovery field condition. After the system receives an event, it queries the alert event based on the recovery field. Events containing the specified field will be automatically recovered. The field that is used to specify the recovery event must be a field in the original event that is equivalent to the alert level. You cannot use the $.severity field. For example, if the recovery field is {$.eventType="resolved"}, the system will automatically recover all alerts under this integration where eventType is resolved.

    2. In the Map Source Fields to Destination Fields area, map the alert source fields to the ARMS alert fields. Click the mapping icon to change the field mapping method.

      • Direct: The specified field from the alert source is directly mapped to the corresponding alert field in ARMS.

      • Concatenate: Multiple alert source fields are concatenated into a single field using a specified separator. This single field is then mapped to the corresponding alert field in ARMS. Only special characters are supported as separators.

      • Conditional: The configured field is mapped to the ARMS destination field only when the alert field value meets the specified condition.

      • Mapping Table: Set the mapping between the alert levels of the alert source and the alert levels in ARMS. A mapping table needs to be set only for the alert level (severity) field.

    ARMS alert field descriptions:

    ARMS Alert Field

    Description

    Alert Name (alertname)

    The name of the custom alert.

    Alert Level (severity)

    The mapping field for the alert level. You must set an alert level mapping table for this field, and the mapping method must be set to Direct.

    Alert Description (message)

    The detailed information of the alert event, used for alert notifications. Supports up to 15,000 characters.

    Alert Sample Value (value)

    The sample value of the monitoring metric.

    Alert Image (imageUrl)

    The URL of the metric line chart, used to map Grafana metric line charts.

    Check Item (check)

    The alert check item. For example: CPU, JVM, Application Crash, Deployment.

    Source (source)

    The source of the alert event.

    Class

    The object type of the alert event, such as host.

    Service (service)

    The business-related source service, such as Login Service.

    Start Time (startat)

    The timestamp of the event start time.

    End Time (endat)

    The timestamp of the event end time.

    Event URL (generatorUrl)

    The URL for the event details.

  7. Configure event deduplication. To reduce duplicate data, the system uses relevant fields as the basis for deduplication. ARMS Alert Management supports previewing the deduplication grouping results of historical event data in the Event Mapping area. You can adjust the deduplication fields as needed.

    Event deduplication applies only to events that are not in a recovered state.

    1. In the Event Deduplication area of the Edit Integration page, select the fields for deduplication. When the values of the selected fields are the same, the corresponding multiple events will be merged into a single alert notification.

    2. Click Deduplication Test to preview the deduplicated alert groups.

    The deduplication test is performed on the last 10 test data entries that are uploaded in the area on the left of the Event Mapping section.

  8. After configuration, click Save.

Integrate self-managed Prometheus alerts

Open source Prometheus supports reporting alerts to Alertmanager. By configuring <alertmanager_config> in the prometheus.yaml file, you can report alerts to the cloud-based alert center. For more information, see [alertmanager_config].

ARMS Alert Management provides an Alertmanager notification capability that is fully compatible with the open source version. By integrating Prometheus, you can report alerts from your self-managed Prometheus to ARMS Alert Management.

  1. Log on to the ARMS console. In the navigation pane on the left, choose Alert Management > Integration.

  2. On the Alert Integration tab, click Prometheus.

  3. In the Create Prometheus Integration dialog box, enter a name and description for the Prometheus integration, and set the automatic alert event recovery time. Then, click Save.

    Automatic alert event recovery time: If an alert event is not triggered again within the specified time, the alert event automatically recovers.

  4. Select the method to connect to Prometheus based on the type of Prometheus instance, and then click OK.

For Prometheus instances managed by Prometheus-Operator

  1. Create an arms-alertmanager-config.yaml file locally and paste the following content into the created file.

    - scheme: https
  bearer_token: ******
  static_configs:
    - targets: ['alerts.aliyuncs.com']

    Obtain the value of bearer_token from the console.

  2. Create a secret named arms-am-config from the arms-alertmanager-config.yaml file.

    kubectl create secret generic arms-am-config --from-file=arms-alertmanager-config.yaml

  3. Create a prometheus-patch.yaml file locally and paste the following content into the created file.

    spec:
  additionalAlertManagerConfigs:
    key: arms-alertmanager-config.yaml
    name: arms-am-config

  4. Execute the following command to retrieve the Prometheus object.

    kubectl get prometheus

    Sample response:

    NAME                                                VERSION   REPLCAS   AGE
prometheus-operator-160879-prometheus               v2.18.2   1         4d1h

  5. Execute the following command to update the Prometheus object with the prometheus-patch.yaml file.

    kubectl patch prometheus prometheus-operator-160879-prometheus --patch "$(cat prometheus-patch.yaml)" --type=merge
For Prometheus instances built in other ways

  1. Modify the prometheus.yaml configuration file.

    The following configuration is for Prometheus 2.35. For configuration formats of other versions, see the Prometheus official documentation.

    alerting:
  alertmanagers:
  - api_version: v1
    authorization:
      type: Bearer
      credentials: ${token}
    scheme: https
    static_configs:
    - targets: ["alerts.aliyuncs.com"]

    Obtain the value of bearer_token from the console.

  2. Access the Prometheus configuration page in your browser to check if the prometheus.yaml file has been modified.

Integrate other Prometheus-like observability systems with ARMS alerts

Thanos uses the --alertmanagers.config parameter of the thanos rule command to specify the alertmanager configuration file. Fill in the integration URL and token as described above. For more information, see the Thanos documentation.

VictoriaMetrics uses the -notifier.config parameter of the ./bin/vmalert command to specify the notifier configuration file. Fill in the integration URL and token as described above. For more information, see the VictoriaMetrics documentation.

Create an event processing flow

After integrating multiple alert sources with ARMS Alert Management, you can set up an event processing flow to filter and classify events generated by the alert sources.

  1. Log on to the ARMS console.

  2. In the navigation pane on the left, choose Alert Management > Event Processing Flow.

  3. On the Event Processing Flow page, click Create Processing Flow in the upper-right corner.

  4. On the Create Processing Flow page, in the Basic Information area, enter a name for the processing flow.

  5. In the Flow Action Configuration area, set up the event processing flow.

    1. Drag actions from the Available Flow Actions area to the Event Processing Flow area.

    2. In the area on the far right, set the execution conditions for each action in the event processing flow.

    After configuring the actions, you can test the alert fields in the Test Data area to verify if the actions are configured correctly.

    Action

    Description

    Example

    Trigger Condition

    Only events that meet the trigger condition will execute the current processing flow.

    Scenario

    An integration named Container CPU Usage Alert executes the current processing flow.

    Action Configuration

    Set the condition to _aliyun_arms_integration_name equals Container CPU Usage Alert.

    Filter Event

    Events that meet the filter condition will skip the current event flow and execute the next one. Unfiltered events will continue to execute the current flow.

    If your alert rule has not yet generated an event, you can select only the preset ARMS fields _aliyun_arms_integration_id and _aliyun_arms_integration_name for configuration. To use event fields such as severity or alertname, you can configure them in one of the following two ways:

    • Manually enter the field.

    • Add an alert rule to generate an alert event. After the event is generated, you can select the event field value here.

    Scenario

    P4 level alert events do not execute the current processing flow.

    Action Configuration

    Set the condition to severity equals P4.

    Identify Event Type

    Identifies the event object classification (Class) and event type (Type) based on the selected event field values using an algorithm. This is used for event and alert query statistics.

    The following event object classifications (Class) are supported:

    • Network

    • Storage

    • Compute

    • Operating System (OS)

    • Application

    • Database

    • Change

    The following event types (Type) are supported:

    • Availability

    • Latency

    • Capacity

    • Error

    • Unknown

    Scenario

    Identify the event object classification (Class) and event type (Type) using the alertname and message field values.

    Action Configuration

    Set Identification Fields to alertname and message.

    Execution Result

    The system writes the intelligent identification results to the class and type labels.

    Set Event Level

    Set the event level for the current processing stream to a specific value.

    Scenario

    Set the level of core service network-type alert events to P1.

    Action Configuration

    1. Set the Trigger Condition action to class equals network.

    2. Set the Set Event Level action to P1.

    Execution Result

    Set Business Label

    Sets a business label (Label) for events that execute the current processing flow. This is used for event and alert statistics queries.

    • Field: Uses the value of a field in the event as the value of the label field.

    • Value: Sets a fixed value for the label field.

    Scenario

    Use the alert level as the alert label to later count events of each level.

    Action Configuration

    Set the business label to Field: severity.

    Execution Result

    A label field will be added to the alert event. The value of the label field will be the value of the severity field.

    Delete Field

    Deletes certain fields from the event.

    Scenario

    The alert event contains location and region fields. The location field already contains region information, so the region field needs to be deleted.

    Action Configuration

    Set the field to delete to region.

    Extract Content

    Extracts the content of a field in an event using a regular expression and populates a new field with the extracted content.

    Scenario

    Alert field:

    cn-hangzhou-hz4

    Extract the region and specific location information from the location field and populate the region and datacenter fields with the extracted information.

    Action Configuration

    • Field to extract from: location

    • Extraction regular expression: ([a-zA-z]+-[a-zA-z]+)-(.*)

    • Fields to fill: region, datacenter

    Execution Result

    Match and Update

    When the system detects a specific piece of content in a specified field of an event, it populates the target field with the specified field value.

    Scenario

    Alert field:

    "message":"ping to i-bp1e42d0ydxf7pstuepz > 100ms"

    When message contains ping, it indicates a network event. The event needs to be marked as a network type.

    Action Configuration

    • Field to match: message

    • Update field value if condition is met: contains ping output network. The contained value can be an actual field value or a regular expression.

    • Fill into field: class

    Execution Result

    Rich fields

    Enriches a field by calling an API or querying a local Excel data source, and populates the target field with the result.

    The data source feature is currently in grayscale release. Contact the Alert DingTalk support account (d9j_rg9e4062f) for assistance.

    Scenario

    Based on the IP field in the alert event, query the uploaded Excel data source and populate the event's hostname field with the resulting hostname.

    Action Configuration

    • Select data source: Get Host Data Source-excel

    • Match and update mode: ip (matches the column name in Excel) equals ip (corresponds to the column value in Excel)

    • Fill into field: $.hostName (column name from the Excel query result) fills into hostname

    Replace Content

    Finds and replaces the content of a specified field. You can use regular expressions to find the content.

    Scenario

    Replace the string d.b. with the string database in the alert event.

    Action Configuration

    • Content snippet to replace: message

    • Replace content snippet: d\.b\. Note: You can use an actual field value or a regular expression.

    • Replace with: database

    Execution Result

    Split Content

    Splits a field using a separator character and populates multiple target fields with the results.

    Scenario

    Alert field:

    "message":"myhostid_myuserid_myruleid"

    Split the value of the message field into hostid, userid, and ruleid, and populate new fields with the results.

    Action Configuration

    • Field to split: message

    • Separator character: _

    • Fields to fill with split results: hostId, userId, ruleId

    The split fields will be filled in order.

    Execution Result

    Fill Template

    Enter template content (for example, ${class}).

    The event source is ${source}.

    ). Variable names are the field names after event integration. The rendered template populates the target field.

    Scenario

    Alert fields:

    "source":"server", "class":"network"

    Set the message field to The event of type ${class} is from source ${source}, which needs to reference the class and source field values.

    Action Configuration

    • Template to fill: The event of type ${class} is from source ${source}

    • Fill into field: message

    Execution Result

    Discard

    Discards events that execute the current processing flow. The discard action can only be the last node of the current event processing flow and does not require configuration.

    None

  6. In the Test Data area on the left, enter test fields and field values, or click Select from Event to select a test event from existing alert events. Then, click Click to Test.

  7. After you configure the settings, click Save. After the event processing flow is created, events from integrated alert sources are filtered in the order defined in the flow. You can view the filtered events on the Alert Event History page. For more information, see View alert event history.

Create alert notification recipients

ARMS Alert Management supports sending alert notifications to contacts by phone call, text message, email, DingTalk, WeCom, Lark, or webhook.

Contacts
Create a contact

  1. Log on to the ARMS console. In the navigation pane on the left, choose Alert Management > Notification Recipient.

  2. On the Contacts tab, click Create Contact.

    In the Create Contact dialog box, set the following parameters as needed, and then click OK.

    Parameter

    Description

    Name

    Enter a custom contact name.

    Mobile Number

    Set the contact's mobile number to receive alert notifications via phone calls and text messages.

    Email

    Set the contact's email address to receive alert notifications via email.

    Contact Group

    Select the contact group to which the contact should be added. For information about how to create a contact group, see Create a contact group.

    Fallback Notification Type for Failed Phone Calls

    Select the fallback notification type if a phone call notification fails.

    User ID

    The user ID for the IM tool used within the enterprise (currently supports DingTalk, Lark, and WeCom), such as a DingTalk ID or WeCom ID.

    This field can be used to @mention contacts in alert cards within a group. To use this feature, enter the correct ID for the corresponding IM tool.

    Currently, the @mention feature for Lark and WeCom depends on this field. DingTalk does not currently depend on this field.

    Note: Only verified mobile phone numbers can be used for the phone call notification method in a notification policy. For more information about how to verify a mobile phone number, see Verify a mobile phone number.

Create a contact group (Optional)

When you create a notification policy, you can specify a contact group as the notification recipient. When the matching rule of the notification policy is triggered, ARMS Alert Management sends alert notifications to all recipients in the contact group by phone call, text message, email, DingTalk, or other methods.

Note: You must create contacts or DingTalk, WeCom, or Lark robots before you can create a contact group.

  1. Log on to the ARMS console. In the navigation pane on the left, choose Alert Management > Notification Recipient.

  2. On the Contacts tab, click Create Contact Group.

  3. In the Create Contact Group dialog box, enter a Group Name, select Alert Contacts, and click OK.

    After creation, the corresponding contact group will be displayed in the list on the left of the Contacts tab.

Alert Robots
DingTalk robot

Before you create a DingTalk robot, you must complete the following steps:

  1. Log on to the ARMS console. In the navigation pane on the left, choose Alert Management > Notification Recipient.

  2. Click the DingTalk/Lark/WeCom tab, and then click DingTalk.

  3. In the panel that appears, set the following parameters, and then click OK.

    Parameter

    Description

    Name

    The name of the DingTalk robot.

    Signature Key

    Optional. If a key is configured, DingTalk authentication will be performed using a signature. If no key is configured, authentication is performed using a keyword whitelist by default. The whitelist keyword is alert. For more information, see the DingTalk official documentation.

    Robot URL

    Enter the webhook URL of the DingTalk robot.

    Robot Sends Daily Statistics

    If selected, you need to enter the time points for sending daily statistics. Use commas (,) to separate multiple time points. The time format is HH:SS. ARMS Alert Management will send the total number of alerts generated, resolved, and pending for the day at the set time points.

    Card Content Configuration

    Customize the style and content of the alert notification card.

WeCom robot

After creating a WeCom robot in ARMS Alert Management, you can specify the corresponding WeCom group in a notification policy to receive alerts. When the matching rule of the notification policy is triggered, the system automatically sends an alert notification to your specified WeCom group. After the WeCom group receives the notification, you can manage the alert within the group.

Before you create a WeCom robot, make sure that you have created a WeCom group to receive alert notifications.

Due to WeCom's licensing restrictions on application service providers, ARMS provides a maximum of 20 WeCom API call licenses. We recommend that you arrange your on-duty staff for handling alerts within WeCom in a reasonable manner. Personnel who do not have a strong need to handle alerts in WeCom are advised to handle alerts in the ARMS console.

Step 1: Obtain the robot webhook URL

  1. Open and log on to WeCom.

  2. Click the icon in the upper-right corner of the WeCom group, and then click Group Robot.

  3. On the Group Robot page, click Add Robot. On the Robot page, click Add.

  4. On the Add Robot page, enter a name for the robot, and then click Add.

  5. Click Copy to save the webhook URL of the WeCom robot.

Step 2: Create the robot in the ARMS console

  1. Log on to the ARMS console. In the navigation pane on the left, choose Alert Management > Notification Recipient.

  2. Click the DingTalk/Lark/WeCom tab, and then click WeCom.

  3. In the create robot panel, set the following parameters, and then click OK.

    Configuration Item

    Description

    Name

    Enter a custom name for the WeCom robot.

    Bot Address

    Enter the webhook URL of the WeCom robot.

    Robot Sends Daily Statistics

    If selected, you need to enter the time points for sending daily statistics. Use commas (,) to separate multiple time points. The time format is HH:SS. ARMS Alert Management will send the total number of alerts generated, resolved, and pending for the day at the set time points.

    Card Content Configuration

    Customize the style and content of the alert notification card.
Lark robot

After creating a Lark robot in ARMS Alert Management, you can specify the corresponding Lark group in a notification policy to receive alerts. When the matching rule of the notification policy is triggered, the system automatically sends an alert notification to your specified Lark group. After the Lark group receives the notification, you can manage the alert within the group.

Step 1: Obtain the robot webhook URL

  1. Open and log on to Lark.

  2. (Optional) Click the + icon, and then click Create Group to create a new group for sending alerts.

  3. In the upper-right corner of the Lark group, choose > Settings, and then click Group Robot.

  4. In the Group Robot panel, click Add Robot, and then select Custom Robot.

  5. On the robot configuration page, set the robot name and description, and then click Add.

  6. Copy the webhook URL, then select Custom Keywords and enter the keyword alert.

Step 2: Create the robot in the ARMS console

  1. Log on to the ARMS console. In the navigation pane on the left, choose Alert Management > Notification Recipient.

  2. Click the DingTalk/Lark/WeCom tab, and then click Lark.

  3. In the create robot panel, set the following parameters, and then click OK.

    Parameter

    Description

    Name

    You can customize the name of the Lark bot.

    Robot URL

    Enter the webhook URL of the Lark robot.

    Does the bot send daily statistics?

    If selected, you need to enter the time points for sending daily statistics. Use commas (,) to separate multiple time points. The time format is HH:SS. ARMS Alert Management will send the total number of alerts generated, resolved, and pending for the day at the set time points.

    Card Content Configuration

    Customize the style and content of the alert notification card.
Customize alert notification recipients via webhook

When you create a notification policy, you can send alert notifications to a custom webhook URL. ARMS Alert Management supports sending webhook alerts to third-party notification recipients. This topic uses Lark as an example to show how to create a webhook alert.

Step 1: Obtain the webhook URL

Configure the Lark webhook robot on a PC.

  1. Open and log on to Lark.

  2. Click the + icon, and then click Create Group to create a new group for sending alerts.

  3. Click the group settings icon, and then click the Group Robot tab.

  4. On the Group Robot tab, click Add Robot.

  5. In the Add Robot panel, select Custom Bot.

  6. On the configuration page, set the Display Name and Description, and then click Add.

  7. In the Add Status area, click Copy, and then click Done.

Step 2: Create a webhook contact

  1. Log on to the ARMS console. In the navigation pane on the left, choose Alert Management > Notification Recipient.

  2. Click the Webhook Integration tab, and then click Create Webhook.

  3. In the Create Webhook dialog box, enter the following configuration information.

    Parameter

    Description

    Webhook Name

    Required. The name of the webhook.

    Post and Get

    Required. Set the request method. The URL cannot exceed 256 characters.

    In this example, select Post and paste the webhook URL saved in Step 1 into the text box on the right.

    Header and Param

    Optional. Set the request header. Cannot exceed 200 characters. Click +Add to add other Header or Param information. The default request header is Content-Type: text/plain; charset=UTF-8. The total number of Headers and Params cannot exceed 6.

    In this example, set the following two Headers:

    • Arms-Content-Type : json

    • Content-Type : application/json

    Notification Template

    The notification template to send when an alert is triggered. This is an optional field that is available only for the Post method. You can use the $content placeholder to output the notification content, which cannot exceed 500 characters. For more information, see Configure notification templates and webhook templates.

    The notification template is as follows:

    {
  "Alert Name":"{{ .commonLabels.alertname }}{{if .commonLabels.clustername }}",
  "Cluster Name":"{{ .commonLabels.clustername }} {{ end }}{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}",
  "Application Name":"{{ .commonLabels._aliyun_arms_involvedObject_name }} {{ end }}",
  "Notification Policy":"{{ .dispatchRuleName }}",
  "Alert Time":"{{ .startTime }}",
  "Alert Content":"{{ for .alerts }} {{ .annotations.message }} {{ end }}"
}

    For Lark, you can set the following text format:

    {
  "msg_type": "text",
  "content": {
    "text": "Alert Name: {{ .commonLabels.alertname }}\n{{if .commonLabels.clustername }}Cluster Name: {{ .commonLabels.clustername }}\n{{ end }}{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}Application Name: {{ .commonLabels._aliyun_arms_involvedObject_name }}\n{{ end }}Notification Policy: {{ .dispatchRuleName }} \nAlert Time: {{ .startTime }} \nAlert Content: {{ for .alerts }} {{ .annotations.message }}\n {{ end }}"
  }
}

    Recovery Template

    Specifies the notification template to send when an alert is recovered. This parameter is optional and is available only for the POST method. You can use the $content placeholder to output the notification content. The template cannot exceed 500 characters. For more information, see Configure notification templates and webhook templates.

    The recovery template is as follows:

    {
  "Alert Name":"{{ .commonLabels.alertname }}{{if .commonLabels.clustername }}",
  "Cluster Name":"{{ .commonLabels.clustername }} {{ end }}{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}",
  "Application Name":"{{ .commonLabels._aliyun_arms_involvedObject_name }} {{ end }}",
  "Notification Policy":"{{ .dispatchRuleName }}",
  "Recovery Time":"{{ .endTime }}",
  "Alert Content":"{{ for .alerts }} {{ .annotations.message }} {{ end }}"
}

    For Lark, you can set the following text format:

    {
  "msg_type": "text",
  "content": {
    "text": "Alert Name: {{ .commonLabels.alertname }}\n{{if .commonLabels.clustername }}Cluster Name: {{ .commonLabels.clustername }}\n{{ end }}{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}Application Name: {{ .commonLabels._aliyun_arms_involvedObject_name }}\n{{ end }}Recovery Time: {{ .startTime }} \nNotification Policy: {{ .dispatchRuleName }} \nRecovered Alert Content: {{ for .alerts }} {{ .annotations.message }}\n {{ end }}"
  }
}

  4. Optional: Click Send Test to verify the configuration.

  5. Click OK.

Notification policy management

Notification policy

By setting a notification policy, you can create matching rules for alert events. When a matching rule is triggered, the system sends an alert message to the notification recipient using your specified notification method. This reminds the recipient to take necessary problem-solving measures.

  1. Log on to the ARMS console. In the navigation pane on the left, choose Alert Management > Notification Policy.

  2. On the Notification Policy page, click Create Notification Policy.

  3. At the top of the Create Notification Policy page, set the name of the notification policy.

  4. In the Matching Rules area, set the matching rules for alert events.

    1. Select the data source.

      1. Specified Source: The notification policy will filter and send notifications for alert events from the specified source (integration).

      2. No Preset Source: The notification policy will filter and send notifications for all alert events.

    2. Set the matching rule expression. You can use custom labels or select existing ones.

      Existing labels include the following:

      1. The fields that are included in an alert rule expression. For more information about how to create labels for alert rules in Managed Service for Prometheus, see Prometheus alerting rules.

      2. Default labels provided by the ARMS system. The default labels are described below.

        Category

        Label

        Description

        Common Fields

        alertname

        Alert name.

        clustername

        Cluster name.

        severity

        Alert level:

        • P1

        • P2

        • P3

        • P4

        • Default

        namespace

        Namespace.

        pod_name

        Pod name.

        System Preset Fields

        _aliyun_arms_integration_name

        Integration name. The default integration name for alerts reported by ARMS is ARMS-DEFAULT.

        _aliyun_arms_involvedObject_id

        Alert object ID.

        _aliyun_arms_involvedObject_name

        Alert object name.

        _aliyun_arms_region_id

        Region ID.

        _aliyun_arms_alert_rule_id

        Alert rule ID.

        _aliyun_arms_alert_type

        Alert rule type:

        • 101: Prometheus alert

        • 5: Application Monitoring alert

        • 4: Browser Monitoring alert

      3. If an alert must meet multiple matching rules simultaneously to be triggered, click Add Condition to edit the second matching rule condition.

      4. If an alert should be triggered when it meets any one of the matching rules, click Add Rule to edit the second matching rule.

    3. Click Next.

  5. In the Event Grouping area, set whether alert events need to be grouped, and then click Next.

    • No Grouping: All alert events will be sent as a single alert message to the handler.

    • Set Grouping Fields: Alert events with the same field content will be aggregated into a single alert notification.

  6. In the Notification Recipient area, set the following parameters.

    1. Click +Add Notification Recipient to select a notification recipient.

      Notification recipient types:

      1. Contact: After selecting a specific contact, you also need to choose the notification method: phone, text message, or email.

      2. Contact Group: After selecting a specific contact group, you also need to choose the notification method: phone, text message, or email.

      3. On-call Schedule: After selecting a specific on-call schedule, you also need to choose the notification method: phone, text message, or email.

      4. DingTalk/Lark/WeCom: Send alert notifications via DingTalk, Lark, or WeCom.

      5. Universal Webhook: Send alert notifications via webhook.

    2. Select whether to send a recovery notification after the alert is resolved.

      Send Recovery Notification: When all events under an alert are recovered, the alert status will automatically change to resolved. When the alert is recovered, the system will send a notification to the alert handler.

    3. Set up notification templates. For more information, see Configure notification templates and Webhook templates.

    4. Set the notification period. Alerts will be sent during the configured notification period.

    5. Optional: Select the ticketing system to which you want to push alerts. For more information about how to integrate a ticketing system, see Notification Integration.

    6. Click Next.

  7. In the Repeat/Escalation/Recovery Policy area, set whether the alert requires repeated notifications or an escalation policy, and then click Next.

    • Repeat Notifications: If repeated notifications are needed, set the repetition frequency. When an alert is not resolved, the alert message will be sent repeatedly at the specified frequency until the alert is resolved.

    • Does the alert have an upgrade policy configured?

      • No Escalation Policy: If the alert is not resolved, it is sent only once.

      • Select Escalation Policy: If the alert is not resolved, the alert notification will be sent to other notification recipients in accordance with the escalation policy.

    • Enable Manual Recovery: If manual recovery is enabled, when an alert event is not triggered again within the automatic recovery time set in the alert integration, the alert does not automatically recover. It must be manually changed to the resolved state.

  8. In the Action Integration area, you can set actions to be automatically executed after an alert is triggered or recovers. For more information, see Execute Alert Plans Through ARMS Action Integration.

  9. After configuration, click Save.

Silence policy

The silence policy in ARMS Alert Management can converge alerts and reduce alert storms. By setting a silence policy, you can create matching rules for alert events. When a rule is triggered, the corresponding event will be silenced and will not be matched by any notification policy.

Silence policies take precedence over notification policies. This means that an alert event matched by a silence policy will be silenced and cannot be matched by a notification policy.

  1. Log on to the ARMS console. In the navigation pane on the left, choose Alert Management > Silence Policy.

  2. On the Silence Policy page, click Create Silence Policy.

  3. On the Create Silence Policy page, set the name of the silence policy.

  4. In the Silence Event Matching Rules area, set the matching rules for silenced events.

  5. Set the Silence Rule Effective Period.

    • Always Active

    • Recurring: Select Daily or Weekly recurrence, then click Add to set the effective time period for the silence rule.

    • Custom Time Period: Click Add to set the effective time period for the silence rule.

  1. After configuration, click Save.

Escalation policy

For alerts that remain unresolved for a long time, you can choose to send an escalation notification to remind contacts to resolve them promptly. After adding an escalation policy to a notification policy, the system will send an alert message to the handler using your specified notification method to remind them to take necessary problem-solving measures.

  1. Log on to the ARMS console. In the navigation pane on the left, choose Alert Management > Escalation Policy.

  2. On the Escalation Policy page, click Create Escalation Policy.

  3. In the text box at the top of the area on the right, enter a name for the escalation policy.

  4. In the escalation rule area, set the notification condition. This means sending an alert notification when an alert has not been claimed or resolved within a specified period. For example: send an escalation notification if an alert is not claimed for 10 minutes.

    Parameter

    Description

    Recipient/On-call Schedule

    Select the notification recipient. For information about how to create a notification recipient, see Create alert notification recipients.

    Notification Method

    Supported notification methods include DingTalk, email, WebHook, text message, and phone call. You can select multiple methods at the same time.

    Note: Contacts with unverified mobile numbers cannot use the phone notification method. To verify a mobile number, see Verify a mobile number.

    Notification Period

    Alerts that meet the notification conditions will be sent during the configured notification period.

    Repetitions

    The number of times to resend the alert notification. When the alert no longer meets the notification conditions, the alert notification will stop being sent.

    Note: The alert is sent at least once. Setting this to 0 will still send the alert once.

  5. After configuration, click Save.

Troubleshooting

Why do I still receive alert notifications even if I selected not to specify a notification policy for the alert?

Regardless of whether a notification policy is specified in the alert rule, the alert is sent to Alert Management. If the alert meets the dispatch conditions of another notification policy, it is sent through that other policy.

Previous：Multi-account full-stack observability dashboard solutionNext：Cost