How to configure a ClickHouse alert rule-ApsaraDB for ClickHouse-阿里云帮助中心

ApsaraDB for ClickHouse lets you set alerts for key cluster metrics. If a metric value crosses a specified threshold, the system sends an alert notification to the contacts in the alert contact group, helping you quickly identify issues. This topic explains how to set up an alert rule for an ApsaraDB for ClickHouse cluster.

Prerequisites

Your cluster must be a Community Compatible Edition or an Enterprise Edition.
Alerting for ApsaraDB for ClickHouse requires Application Real-Time Monitoring Service (ARMS). RAM users must have the AliyunARMSFullAccess permission. For details on how to grant this permission, see Manage permissions for a RAM user.

Notes

CloudMonitor does not currently support monitoring for Enterprise Edition clusters.

Steps

ApsaraDB for ClickHouse console

Note

If your cluster meets both of the following conditions, see Operations in the previous console.

The cluster was created before December 1, 2021.
The cluster is in the China (Qingdao) or China (Hohhot) region.

Log on to the ApsaraDB for ClickHouse console.
In the upper-left corner of the page, select the region where the target cluster is located.
On the Clusters page, click the Clusters of Community-compatible Edition or Enterprise Edition Clusters tab, and then click the ID of the target cluster.
In the left-side navigation pane, click Monitoring and Alerting.
On the Monitoring Details page, click the Cluster Alerting tab.
Click the Create cluster alert rule button.

Note
For Community Edition clusters, this button is Create ClickHouse Alert Rule.

For Enterprise Edition clusters, this button is Create Enterprise Edition ClickHouse Alert Rule.

On the Create ClickHouse alert rule or Create Enterprise Edition ClickHouse alert rule page, set the following parameters.

Note

ApsaraDB for ClickHouse alert rules support two detection types: static threshold and custom PromQL. The parameters vary based on the selected detection type.

Use the static threshold type to create an alert rule quickly by selecting a preset metric.
Use the custom PromQL detection type to create alert rules for metrics not preset by the system.

Static threshold

Parameter	Description		Example
Alert name	The name of the alert rule.		CPU utilization alert
Detection type	Select static threshold.		Static threshold
Cluster	Select the target cluster.		cc-bp1lxbo89u95****
Alert group	Select an alert group. The available alert groups depend on the Prometheus instance type.		ClickHouse Community Edition
Alert metric	Select the alert metric. Available metrics depend on the selected alert group.		CPU utilization
Alert condition	Configure the condition that triggers the alert using the options provided for the selected metric.		An alert is triggered when the CPU utilization is `greater than` 80%.
Filter condition	No filter		None
Data preview	The Data preview section displays the PromQL statement that corresponds to the alert condition and shows the values of the monitored metric as a time-series chart. By default, this section displays the real-time value for a single resource. You can select a target resource and a time range in this section to view values for different resources and time periods. Note A red horizontal line on the time-series chart indicates the alert threshold. The part of the curve that triggers an alert appears in dark red, while the rest appears in blue. You can hover over the time-series chart to view resource details at a specific point in time. You can select a time range on the chart to zoom in on the time-series curve for that period.		N/A
Duration	An alert is triggered as soon as a data point meets the alert condition. An alert is triggered only if the alert condition persists for a specified number of minutes.		1
Alert level	The severity level of the alert. Levels, in increasing order of severity, are: Default, P4, P3, P2, and P1.		P2
Alert content	The content of the alert notification. You can use Go template syntax to customize the content with variables.		`Node: {{$labels.pod_name}} CPU utilization {{$labels.metrics_params_opt_label_value}} {{$labels.metrics_params_value}}%, current value: {{ printf "%.2f" $value }}%`
Alert notification	Simple mode	Alert Contact: The contacts who will receive alert notifications. Notification period: The time period during which notifications are sent. Alerts triggered outside this period are ignored. Repeat policy: Defines how often to resend notifications for an ongoing alert.
	Normal mode	Specify a quick dispatch policy. Important A Quick dispatch policy links alert events from this rule to a specific notification policy. However, other policies that use fuzzy matching might also match the same events, which can result in duplicate notifications. Do not specify a notification policy. Select an existing notification policy. Note You can click Create notification policy to create a new notification policy. For more information, see Notification policies.	Do not specify a notification policy
Advanced settings
Check interval	How frequently the alert condition is checked. The minimum and default interval is 1 minute.		1 minute
Check after data is complete	Waits until the collected data is fully transmitted and stored before checking for alerts. This option is enabled by default and should typically not be changed.		Yes
Tag	Labels to attach to the alert. These can be used as matching criteria in notification policies.		None
Annotation	Annotations to attach to the alert.		None

Custom PromQL

Parameter	Description		Example
Alert name	The name of the alert rule.		Pod CPU utilization greater than 8%
Detection type	Set to custom PromQL.		Custom PromQL
Cluster	Select the target cluster.		cc-bp1lxbo89u95****
Reference alert group	Select an alert group. The available alert groups depend on the Prometheus instance type.		ClickHouse Community Edition
Reference alert metric	Optional. Select a similar metric to pre-fill the PromQL configuration, which you can then modify. The available reference metrics are automatically filtered based on the selected Prometheus instance type.		Number of HTTP Connections
Custom PromQL statement	Use a PromQL expression to define the alert rule.		`clickhouse_http_conn_usage_count{} > 1000`
Data preview
Duration	An alert is triggered as soon as a data point meets the alert condition. An alert is triggered only if the alert condition persists for a specified number of minutes.		1
Alert level	The severity level of the alert. Levels, in increasing order of severity, are: Default, P4, P3, P2, and P1.		Default
Alert content	The content of the alert notification. You can use Go template syntax to customize the content with variables.		`Namespace: {{$labels.namespace}}/Pod: {{$labels.pod_name}}/Disk device: {{$labels.device}} utilization exceeds 90%, current value: {{ printf "%.2f" $value }}%`
Alert notification	Simple mode	Alert Contact: The contacts who will receive alert notifications. Notification period: The time period during which notifications are sent. Alerts triggered outside this period are ignored. Repeat policy: Defines how often to resend notifications for an ongoing alert.
	Normal mode	Specify a quick dispatch policy. Important A Quick dispatch policy links alert events from this rule to a specific notification policy. However, other policies that use fuzzy matching might also match the same events, which can result in duplicate notifications. Do not specify a notification policy. Select an existing notification policy. Note You can click Create notification policy to create a new notification policy. For more information, see Notification policies.	Do not specify a notification policy
Advanced settings
Check interval	How frequently the alert condition is checked. The minimum and default interval is 1 minute.		1 minute
Check after data is complete	Waits until the collected data is fully transmitted and stored before checking for alerts. This option is enabled by default and should typically not be changed.		Yes
Tag	Labels to attach to the alert. These can be used as matching criteria in notification policies.		None
Annotation	Annotations to attach to the alert.		None

After you complete the configuration, click Completed.

Use the CloudMonitor console

Log on to the CloudMonitor console.
In the left-side navigation pane, choose Alarms > Alert Rules.
On the Alert Rules page, click Create Alert Rule.

In the Create Alert Rule panel, configure the following parameters.

Parameter		Description
Product		Select the cluster type for ApsaraDB for ClickHouse. Valid values: ApsaraDB for ClickHouse: For ApsaraDB for ClickHouse Community-Compatible Edition clusters purchased before December 1, 2021. ApsaraDB for ClickHouse Community-Compatible Edition: For ApsaraDB for ClickHouse Community-Compatible Edition clusters purchased after December 1, 2021.
Resource Range		The resources that the alarm rule monitors. Valid values: All Resources: The alarm rule applies to all resources of the selected cloud product, including newly added resources. Application Group: The alarm rule applies to all resources within a specified application group for the selected cloud product, including newly added resources. Instance: The alarm rule applies only to specific instances of the selected cloud product.
Rule description		Defines the conditions that trigger an alarm. An alarm is triggered when monitored data meets these conditions. To configure the conditions: Click Add Rule. In the Set Rule panel, enter a Rule Name and then configure the rule conditions. Single Metric: Select a metric, and then set a threshold and an alarm level for it. Multiple Metrics: Select an alarm level, and then set alarm conditions for two or more metrics. Intelligent Threshold: For more information about intelligent thresholds, see Overview and Create an alarm rule for an intelligent threshold. Click OK. Note For information about how to set complex alarm conditions, see Expressions of alarm rules.
Mute For		The time to wait before resending a notification for an unresolved alarm. Valid values: 5 minutes, 15 minutes, 30 minutes, 60 minutes, 3 hours, 6 hours, 12 hours, and 24 hours. After an alarm is triggered, CloudMonitor sends a notification. If the alarm persists, CloudMonitor waits for the Mute For period to pass before sending another notification. No notifications are sent during this period. For example, if you set Mute For to 12 hours, CloudMonitor sends another alarm notification after 12 hours if the alarm is not resolved.
Effective period		The period during which the alarm rule is active. Alarm notifications are sent only during this period. Note No notifications are sent when an alarm rule is outside its effective period. However, the Alarm History list still records these alarms.
Alert contact group		The group of contacts who receive alarm notifications. An alert contact group can contain one or more alert contacts. For more information about how to create alert contacts and alert contact groups, see Create an alert contact or an alert contact group.
Tag		The tags of the alarm rule. A tag consists of a key and a value. Note You can add up to six tags to a rule.
Advanced settings	Alert callback	A publicly accessible URL to which CloudMonitor sends alarm information using POST requests. Only the HTTP protocol is supported. For more information about how to configure an alert callback, see Use alert callbacks for threshold-triggered alarms. To test the callback URL: Click Test next to the callback URL. In the Webhook Test panel, you can check the connectivity of the callback URL and troubleshoot issues based on the returned status code and test result details. Note You can also set the Callback Template Type and Language for the webhook, and click Test again to view the corresponding test result details. Click Disable.
	Auto Scaling	If you enable Auto Scaling, an alarm triggers the corresponding scaling rule. You must specify the Region, scaling group, and scaling rule for Auto Scaling. For more information about how to create a scaling group, see Configure a scaling group. For more information about how to create a scaling rule, see Configure a scaling rule.
	Log Service	If you enable Log Service, CloudMonitor sends alarm information to a Logstore in Log Service when an alarm occurs. You must specify the Region, Project, and Logstore for Log Service. For more information about how to create a Project and a Logstore, see Use Logtail to collect and analyze text logs from ECS instances.
	Message Service (MNS) — topic	If you enable the Message Service (MNS) — topic switch, CloudMonitor sends alarm information to a topic in Message Service (MNS) when an alarm occurs. To enable this feature, you must specify the region and topic for Message Service (MNS). For more information about how to create a topic, see Create a topic.
	Function Compute	If you enable Function Compute, CloudMonitor sends alarm notifications to Function Compute for processing when an alarm occurs. You must specify the region, service, and function for Function Compute. For more information about how to create a service and a function, see Create a function.
	Method for processing missing data	Specifies how to handle alarms when monitoring data is missing. Valid values: Do Not Process (default) Send No-data Alarm Considered Restored

(Optional) Old console

Log on to the ApsaraDB for ClickHouse console.
In the upper-left corner of the page, select the region where the target cluster is located.
On the Clusters page, click the default instance list tab, and then click the target cluster ID.
In the left-side navigation pane, click Monitoring Details.
In the upper-right navigation bar, click Alert Monitoring.
In the CloudMonitor console, choose alert service > alert rule in the left-side navigation pane.
On the threshold alert tab, click Create alert rule.

On the Create alert rule page, configure the following parameters.

Set associated resources. The following table describes the parameters.

Parameter	Description
Product	Select ApsaraDB for ClickHouse from the drop-down list.
Resource Range	All Resources: An alert is triggered if any cluster of the selected product meets the conditions of the alert rule. Cluster: An alert is triggered only if a selected cluster meets the conditions of the alert rule.
Region	This parameter is required if Resource Range is set to Cluster. Select the region where the cluster is located.
Cluster	This parameter is required if Resource Range is set to Cluster. Select one or more cluster IDs.

Set the alert rule. For more information, see Create an alert rule.
Set the alert notification method.

Note
You must first create an alert contact group. For more information, see Create an alert contact or an alert contact group.

After you configure the parameters, click OK. The alert rule is enabled automatically.

Next steps

For information on how to manage the alert rules you create in the ClickHouse console, see alert rule management.

References

For an introduction to Cloud Monitor, see What is Cloud Monitor.