DocsICP FilingConsole
Documentation
Home Page Application Real-Time Monitoring Service Alert Management Use Cases Configure notification templates and webhook templates

Configure notification templates and webhook templates

Updated at:
Copy as MD
Product
My Favorites

When an alert rule in a notification policy is triggered, ARMS sends alert notifications through channels such as DingTalk, Lark, WeCom, SMS, phone, email, and webhooks. ARMS provides pre-configured notification templates for each notification method. If these default templates do not meet your needs, you can customize them. This topic explains the Go template syntax for customization, describes the pre-configured templates, and lists the available template variables.

Go template syntax

ARMS alert notification templates use the Go template syntax to populate the template with data from a provided context.

Note

Alert parameters in a notification template consist of multiple labels. Labels allow you to include specific alert parameters in your notifications. For example, the alertname label in this example indicates the alert name.

This topic uses the following sample code to describe common Go template syntax.

{
  "alerts": [
    {
      "annotations": {
        "message": "Namespace: arms-prom / Pod: kube-state-metrics-ccb59dbff-jljg4 / Container: kube-state-metrics memory usage has exceeded 80%, current value 15.52%",
        "value": "15.521240234375"
      },
      "endsAt": "2023-02-22T07:27:15.404000000Z",
      "fingerprint": "bec72890cc2c7b4a027e008df0cd1013",
      "labels": {
        "container": "kube-state-metrics",
        "severity": "warning",
        "instance": "10.0.80.186:10255",
        "clustername": "klyz1688-kubernetes-1",
        "alertname": "Container memory usage exceeds 80%",
        "_aliyun_arms_involvedObject_name": "klyz1688-kubernetes-1",
        "pod_name": "kube-state-metrics-ccb59dbff-jljg4",
        "_aliyun_arms_involvedObject_kind": "cluster",
        "name": "k8s_kube-state-metrics_kube-state-metrics-ccb59dbff-jljg4_arms-prom_359508f3-7e76-4740-b915-41ea48849641_0",
        "namespace": "arms-prom"
      },
      "startsAt": "2023-02-22T07:18:15.578000000Z",
      "status": "firing"
    }
  ],
  "commonAnnotations": {
    "message": "Namespace: arms-prom / Pod: kube-state-metrics-ccb59dbff-jljg4 / Container: kube-state-metrics memory usage has exceeded 80%, current value 15.52%",
    "value": "15.521240234375"
  },
  "commonLabels": {
    "container": "kube-state-metrics",
    "severity": "warning",
    "instance": "10.0.80.186:10255",
    "clustername": "klyz1688-kubernetes-1",
    "alertname": "Container memory usage exceeds 80%",
    "_aliyun_arms_involvedObject_name": "klyz1688-kubernetes-1",
    "pod_name": "kube-state-metrics-ccb59dbff-jljg4",
    "_aliyun_arms_involvedObject_kind": "cluster",
    "name": "k8s_kube-state-metrics_kube-state-metrics-ccb59dbff-jljg4_arms-prom_359508f3-7e76-4740-b915-41ea48849641_0",
    "namespace": "arms-prom"
  },
  "groupLabels": {
    "alertname": "Container memory usage exceeds 80%"
  },
  "status": "firing",
  "startTime":"2023-02-22 07:18:15",
  "endTime":"Alert resolution time",
  "level":"error",
  "dispatchRuleName":"notification policy name",
  "alarmId":"123456"
},

The . character

Represents the current object in scope, allowing you to access its data.

Example 1: Get all content from the top-level scope, which is the entire context data from the sample code.

{{ . }}

Returns the entire content of the sample code.

Example 2: Get the level field from the context data.

{{ .level }}

Returns:

error

Example 3: Get the commonLabels.alertname field from the context data.

{{ .commonLabels.alertname }}

Returns:

Container memory usage exceeds 80%

Variables

You can store data passed to the template in variables that are accessible throughout the template. For example, in {{$context := .}}, the $context variable stores the incoming data, which you can access using {{$context}}.

Example 1: Assign the alert name to $alertname, and then output $alertname.

{{ $alertname := .commonLabels.alertname }} {{$alertname}}

Returns:

Container memory usage exceeds 80%

Example 2: Assign the first alert from the alerts list to $alert0 and output its alertname.

{{ $alert0 := index .alerts 0 }} {{$alert0.labels.alertname}}

Returns:

Container memory usage exceeds 80%

The if/else action

The if action provides conditional logic. If the condition is false, an optional else block is executed. Values like 0, nil, or an empty string are evaluated as false.

Example 1: If the value of the severity label in an alert event is warning, output P3 alert.

{{if eq "warning" .commonLabels.severity }} P3 alert {{ end }}

Returns:

P3 alert

Example 2: If the value of the severity label in an alert event is critical, output P1 alert. Otherwise, output P2 alert.

{{if eq "critical" .commonLabels.severity }} P1 alert {{ else }} P2 alert {{ end }}

Returns:

P2 alert

The for action

The for action executes a block of code repeatedly.

Example 1: Iterate through the alerts list to get the alertname of each alert.

{{ for .alerts}} {{.labels.alertname}} \n {{end}}

Returns:

Container memory usage exceeds 80%

The range action

The range action iterates over a data structure, such as a slice or map.

Example 1: Iterate over a list to get the index and value of each element.

{{ range $index, $alert := .alerts }}Index {{ $index }}: {{ $alert.labels.alertname }}\n{{ end }}

Returns:

Index 0: Container memory usage exceeds 80%

The index function

The index function accesses an element of an array, slice, or map.

Example 1: Get the first element from the alerts list and assign it to $alert0.

{{ $alert0 := index .alerts 0 }} {{$alert0.labels.alertname}}

Returns:

Container memory usage exceeds 80%

Comparison functions

The following comparison functions are commonly used:

  • eq: Equal to

  • ne: Not equal to

  • lt: Less than

  • le: Less than or equal to

  • gt: Greater than

  • ge: Greater than or equal to

Example: If the value of the severity label in commonLabels is equal to warning, output P3 alert.

{{if eq "warning" .commonLabels.severity }} P3 alert {{ end }}

Returns:

P3 alert

Logical functions: and, or, not

Example: If the value of the severity label in commonLabels is equal to warning or warnning, output P3 alert.

{{if or (eq "warning" .commonLabels.severity) (eq "warnning" .commonLabels.severity) }} P3 alert {{ end }}

Returns:

P3 alert

Common built-in functions

printf: Formats a string according to a specified format specifier.

Example: Output the value of the first alert in the alerts list, formatted to two decimal places.

{{ $alert0 := index .alerts 0 }} {{ printf "%.2f" $alert0.annotations.value }}

Returns:

15.52

humanizePercentage: Formats a value as a percentage.

Example: Convert the string "0.95332123123124" to a more readable percentage.

{{ "0.95332123123124" | humanizePercentage }}

Returns:

95.33%

humanizeDate: Formats a millisecond timestamp into a date. The date format is yyyy-MM-dd HH:mm:ss.

Example: Convert the timestamp 1671074058234 to a date.

{{ "1671074058234" | humanizeDate }}

Returns:

2022-12-15 11:14:18

len: Returns the length of an array, slice, string, or map.

Example: Get the length of the alerts list.

{{ len .alerts }}

Returns:

1

Pre-configured notification templates

Email notification template

Alert name: {{ .commonLabels.alertname }}
{{if .commonLabels.clustername }}
Cluster name: {{ .commonLabels.clustername }}
{{ end }}
{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}
Application name: {{ .commonLabels._aliyun_arms_involvedObject_name }}
{{ end }}
Notification policy: {{ .dispatchRuleName }}
Alert time: {{ .startTime }}
Alert content: {{ if .newIncidentIn }} (New event) {{end}} {{ for .alerts }} {{ .annotations.message }} {{if .generatorURL }}  <a href="{{.generatorURL}}" >Details URL</a> {{ end }} {{ end }}

Template content

Description

 
Alert name: {{ .commonLabels.alertname }}

The alert name. By default, a notification policy groups alerts by the alertname label, so all events in a notification have the same alert name.

 
{{if .commonLabels.clustername }} Cluster name: {{ .commonLabels.clustername }}
                {{ end }}

If an event contains the clustername label, the clustername label value will be rendered as the cluster name field.

 
{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }} Application name: {{ .commonLabels._aliyun_arms_involvedObject_name }}
                {{ end }}

If the involved object's kind is app, the application name is included.

 
Notification policy: {{ .dispatchRuleName }}

The name of the matching notification policy.

 
Alert time: {{ .startTime }}

The time when the alert was first triggered.

 
Alert content: {{ if .newIncidentIn }} (New event) {{end}} 
{{ for .alerts }} {{ .annotations.message }} {{if .generatorURL }}  <a href="{{.generatorURL}}" > Details URL</a> {{ end }} {{ end }}

Iterate over the list of alert events to render the message value. If an event contains the generatorUrl field, render the value of the generatorUrl field as the event details URL.

Note

The pre-configured template uses HTML <a></a> syntax to render the link.

<a href="{{.generatorURL}}">Details URL</a>

The {{ if .newIncidentIn }} (New event) {{end}} block marks the notification by outputting New event when a new event is added to an alert group.

Example of rendered output:

Alert name: Container memory usage exceeds 80%
Cluster name: klyz1688-kubernetes-1
Notification policy: notification policy name
Alert time: 2023-02-22 07:18:15
Alert content: Namespace: arms-prom / Pod: kube-state-metrics-ccb59dbff-jljg4 / Container: kube-state-metrics memory usage has exceeded 80%, current value 15.52%

SMS and phone notification templates

These templates are similar to the email template but omit the details URL.

{{ .level }} alert occurred
Alert name: {{ .commonLabels.alertname }}
{{if .commonLabels.clustername }}
Cluster name: {{ .commonLabels.clustername }}
{{ end }}
{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}
Application name: {{ .commonLabels._aliyun_arms_involvedObject_name }}
{{ end }}
Notification policy: {{ .dispatchRuleName }}
Alert time: {{ .startTime }}
Alert content: {{ if .newIncidentIn }} (New event)
 {{end}} {{ for .alerts }} {{ .annotations.message }} {{ end }}

Chatbot notification template

{{ if .newIncidentIn }} (New event) {{end}}
        {{if .commonLabels.clustername }} > Cluster name: {{ .commonLabels.clustername }}
        {{ end }}
        {{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }} > Application name: {{ .commonLabels._aliyun_arms_involvedObject_name }}
        {{ end }}
        {{ for .alerts }} > {{ .annotations.message }} {{if .generatorURL }} [details URL]({{.generatorURL}}) {{end}}{{if .annotations._aliyun_arms_insights_analyze_link }}[<font color='#ff0000'>root cause</font>]({{ .annotations._aliyun_arms_insights_analyze_link}}){{ end }}{{if eq "1" .labels._aliyun_arms_denoise_code }} (Important:{{.labels._aliyun_arms_important_reason }}) {{end}}
        {{end}}

The output is rendered in Markdown. Feature support varies by chatbot.

  • Blockquote: >

  • Font color:

    <font color='#ff0000'>root cause</font>
    Note

    Only DingTalk and WeCom support font color changes. Lark does not.

  • Line breaks:

    • DingTalk: Add two newlines in the template.

    • Lark: Add one newline in the template.

    • WeCom: Add one newline in the template.

Webhook template

{
  "alert_name":"{{ .commonLabels.alertname }}",
  {{if .commonLabels.clustername }}"cluster_name":"{{ .commonLabels.clustername }}",{{ end }}
  {{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}"application_name":"{{ .commonLabels._aliyun_arms_involvedObject_name }}",{{ end }}
  "notification_policy":"{{ .dispatchRuleName }}",
  "alert_time":"{{ .startTime }}",
  "alert_content":"{{ range $index, $alert := .alerts }}{{if $index}}, {{end}}{{.annotations.message}}{{ end }}"
}

Example of rendered output:

{
  "alert_name": "Container memory usage exceeds 80%",
  "cluster_name": "klyz1688-kubernetes-1",
  "notification_policy": "notification policy name",
  "alert_time": "2023-02-22 07:18:15",
  "alert_content": "Namespace: arms-prom / Pod: kube-state-metrics-ccb59dbff-jljg4 / Container: kube-state-metrics memory usage has exceeded 80%, current value 15.52%"
}

Variable reference

The following table describes the variables available in notification templates.

Parameter

Description

alarmId

The ID of the original alert.

alerts

A list of the original alert events. Type: List<Alert>.

alerts.annotations

The annotations of an original alert. Type: map of strings.

alerts.endsAt

The resolution time of the original alert.

Resolution time = Trigger time + Resolution duration

alerts.startsAt

The trigger time of the original alert.

alerts.fingerprint

The fingerprint of the original alert. Alerts with the same set of labels have the same fingerprint.

alerts.labels

The labels of the original alert.

alerts.status

The status of an original alert:

  • firing: The alert is active.

  • resolved: The alert is resolved.

commonAnnotations

A set of annotations that are common to all alerts in the group.

commonLabels

A set of labels that are common to all alerts in the group.

groupLabels

The labels used to group the alerts, as defined in the notification policy.

status

The status of the alert group:

  • firing: The alert is active.

  • resolved: The alert is resolved.

startTime

The time when the alert group was created.

endTime

The time when the alert group was resolved. This is the resolution time of the last original alert in the group.

level

The severity level of the notification. The values map to priority levels:

  • critical: P1

  • error: P2

  • warning: P3

  • page: P4

dispatchRuleName

The name of the matching notification policy.
Previous：Notification policy best practicesNext：ARMS unified alert management