Reference log content in alert notifications

For example, to monitor NGINX access errors, run the following query to get the HTTP status code, the source IP address, and the error count. Then create an alert monitoring rule and reference this data in the alert template. When a status code reaches 400 or above, SLS sends a notification that includes the status code, source IP, and count.

• Query statement

  status >=400 | SELECT status, __source__ AS ip, count(*) AS cnt GROUP BY status,ip

• Query results

Use the annotations variable

Add the annotations variable to an alert template to include annotation key-value pairs in alert notifications.

• Alert monitoring rule

  Add annotation key-value pairs manually, or turn on Auto-Add Annotations to let SLS populate them automatically when the alert fires. For more information, see Annotations.

  Note

  SLS provides two annotations by default.

  For example, to include the status code, source IP, and error count in alert messages, add annotations such as status:${status}, ip:${ip}, and cnt:${cnt}.

  • Manually add information about annotations

  • Turn on Auto-Add Annotations to automatically add information about annotations

• Alert message

  After an alert fires, SLS generates an alert message containing the annotations you configured:

  {
      "annotations": {
          "title": "Alert is triggered by an NGINX access error",
          "desc": "The error that is indicated by the status code 400 occurred 15 times"
          "status": "400"
          "ip": "127.0.0.1"
          "cnt": "15"
          "__count__": "1"
      }
  }

• Alert template

  Add the annotations variable to the alert template to reference specific log content in notifications. For more information about creating an alert template, see Create an alert template.

  

Use the fire_results variable

Add the fire_results variable to an alert template to list every data row that triggered the alert.

• Alert monitoring rule

  SLS generates alert messages based on the alert monitoring rule you create.

• Alert message

  After an alert fires, SLS generates an alert message containing all rows that matched the trigger condition — in this example, the status code, source IP, and error count for each matching row.

  {
      "fire_results": [
          { "status": "401", "ip": "127.0.0.1", "cnt": "3" },
          { "status": "400", "ip": "127.0.0.1", "cnt": "7" },
          { "status": "501", "ip": "127.0.0.1", "cnt": "4" },
          { "status": "404", "ip": "127.0.0.1", "cnt": "4" },
          { "status": "402", "ip": "127.0.0.1", "cnt": "6" },
          null
      ]
  }

• Alert template

  Add the fire_results variable to the alert template to reference specific log content in notifications. For more information about creating an alert template, see Create an alert template.

  

Use the results variable

Add the results variable to an alert template to include full query context in alert notifications.

• Alert monitoring rule

  SLS generates alert messages based on the alert monitoring rule you create.

• Alert message

  After an alert fires, SLS generates an alert message containing the full query context.

  Note

  If an alert monitoring rule contains multiple query statements, the results array contains multiple items. Each item corresponds to a query statement.

  {
      "results": [{
          "store_type": "log",
          "region": "cn-hangzhou",
          "project": "test-alert",
          "store": "nginx-access-log",
          "query": "status >= 400 | select status, __source__ as ip, count(*) as cnt group by status, ip",
          "start_time": 1640006894,
          "end_time": 1640007014,
          "dashboard_id": "",
          "raw_results": [
              { "status": "401", "ip": "127.0.0.1", "cnt": "3" },
              { "status": "400", "ip": "127.0.0.1", "cnt": "7" },
              { "status": "501", "ip": "127.0.0.1", "cnt": "4" },
              { "status": "404", "ip": "127.0.0.1", "cnt": "4" },
              { "status": "402", "ip": "127.0.0.1", "cnt": "6" },
              null
          ],
          "raw_result_count": 6,
          "fire_result": {
              "status": "401",
              "ip": "127.0.0.1",
              "cnt": "3"
          },
          "has_sql": true,
          "truncated": false,
          "role_arn": ""
      }]
  }

  Each item in the results array contains the following fields:

  Field	Description
  store_type	The type of data store queried, such as log
  region	The region of the project
  project	The SLS project name
  store	The Logstore name
  query	The query statement that was executed
  start_time	The query start time (Unix timestamp)
  end_time	The query end time (Unix timestamp)
  dashboard_id	The associated dashboard ID, if any
  raw_results	All rows returned by the query
  raw_result_count	The total number of rows in raw_results
  fire_result	The specific row that triggered the alert
  has_sql	Whether the query contains a SQL analysis statement
  truncated	Whether the result set was truncated
  role_arn	The RAM role ARN used for the query, if any

• Alert template

  Add the results variable to the alert template to reference specific log content in notifications. For more information about creating an alert template, see Create an alert template.

  

Template variables

{%- for result in alert.fire_results %}
- status: {{ result.status }}, count: {{ result.cnt }}
{%- endfor %}

SLS provides built-in functions for formatting notification content. For more information, see Syntax for new alert templates and Built-in template functions.