Log alerting

Prerequisites

• The advanced monitoring and alert service is available in the following regions: Hangzhou, Beijing, Shanghai, Shenzhen, Qingdao, Zhangjiakou, Chengdu, Guangzhou, Ulanqab, Beijing Finance Cloud, US (Virginia), US (Silicon Valley), Japan (Tokyo), Indonesia (Jakarta), and China (Hong Kong). The regions available in the console are definitive.

• Create a DingTalk robot and configure it with the keywords OK and Alerting. Then, obtain the robot's webhook. For more information, see Receive alert notifications through a DingTalk group.

• Familiarize yourself with Grafana monitoring dashboards. For more information, see Grafana Dashboards.

Limits

• Log alert notifications can be sent only through DingTalk and webhook.

• The custom keywords for the DingTalk robot must be OK and Alerting. Otherwise, you cannot receive alert notifications. The following figure shows an example.

• Grafana alerts cannot be configured for metric data.

Procedure

1. Step 1: Go to the Advanced Monitoring and Alerting page

2. Step 2: Configure a log alert contact

3. Step 3: Configure a log alert rule

4. Step 4: View the log alert rule

Step 1: Go to the Advanced Monitoring and Alerting page

1. Log on to the Alibaba Cloud Elasticsearch console.

2. In the navigation pane on the left, click Advanced Monitoring and Alerting.

Step 2: Configure information for the log alert contact

1. On the Advanced Monitoring and Alerting page, in the navigation pane on the left, choose Monitoring Visualization > Log Monitoring.

2. Click Configure Log Alerting to open the Grafana page.

3. In the navigation pane on the left, click the icon and choose Notification channels.

   

4. Click Add channel and configure the notification channel.

   Parameter	Description
   Name	The custom alert name. In this example, the value is test.
   Type	Only DingTalk and Webhook are supported. Other methods are not supported. In this example, the value is DingDing.
   Url	The webhook path of the DingTalk robot.
   Message Type	The message type. Only Link is supported.

   

5. Click Save to save the configuration.

Step 3: Configure a log alert rule

1. On the Advanced Monitoring and Alerting page, in the navigation pane on the left, choose Monitoring Visualization > Log Monitoring.

2. Click Configure Log Alerting to open the Grafana page.

   

3. Configure a log alert rule using an alert template. You cannot directly edit log alert templates.

   1. On the Grafana page, click the icon on the monitoring template.

   2. In the dialog box that appears, click the Export tab.

      

   3. Click View JSON.

   4. Click Copy to Clipboard to copy the JSON template.

   5. In the navigation pane on the left, click the icon and choose Import.

   6. In the Import via Panel JSON section, paste the default alert rule from the JSON template.

      

   7. Click Load.

   8. Change the Name and redefine the Unique identifier (uid).

   9. Click Import to import the template.

      

4. Adjust the alert rule.

   The following example shows how to configure an alert for the Number of slow queries that exceed the time threshold.

   1. Click Number of slow queries that exceed the time threshold and choose Edit from the drop-down list.

      

   2. Click the Query tab and set the query conditions. The alert is triggered when these conditions are met.

      

   3. Click the Alert tab and set the Rule and Conditions.

      

      Note

      • The default alert rule is as follows: The query condition is executed every 5 minutes. An alert is triggered if the average number of logs for slow queries that take more than 1000 ms is greater than 10 for each data point within the last 5 minutes, and this state persists for more than 1 minute.

      • Evaluate every specifies the detection frequency. The minimum value is 1 minute. Other configuration items do not have specific limits.

   4. Click the icon next to Send to to add the log alert contact that you configured in Step 2.

      

   5. Click Save.

   6. Click Apply.

   Note

   For configuration items that are not mentioned, you can use the default values or configure them as needed. For more information about advanced alert configurations, see the official Grafana documentation.

Step 4: View the log alert rule

1. On the Advanced Monitoring and Alerting page, in the navigation pane on the left, choose Monitoring Visualization > Log Monitoring.

2. Click Configure Log Alerting to open the Grafana page.

3. In the navigation pane on the left, click the icon and choose Alert Rules to view the list of alert rules.