Configure alert rules
ApsaraDB for SelectDB integrates the alerting functionality of Alibaba Cloud Application Real-Time Monitoring Service (ARMS). In the SelectDB console, you can configure alert rules. You can configure monitoring items. When an alert rule for a monitoring item is triggered, all contacts in the alert contact group are notified. You can also maintain alert contact groups for monitoring items to ensure that relevant contacts are promptly notified when an alert is triggered. By configuring alert rules for important monitoring metrics, you can be promptly notified when the metric data of a SelectDB instance becomes abnormal, allowing you to quickly identify and troubleshoot faults. This topic describes how to configure alert rules in the SelectDB console.
You can also configure alert rules for ApsaraDB for SelectDB instances in the CloudMonitor console. For more information, see Cloud service monitoring.
Prerequisites
An ApsaraDB for SelectDB service-linked role, AliyunServiceRoleForSelectDB, must be created. By default, this role has permission to access the ARMS service. For more information, see Service-linked role.
If you want to report monitoring data from your ApsaraDB for SelectDB instance to your own ARMS for centralized monitoring and alerting, you must first activate the ARMS service. For more information, see Activate ARMS.
Procedure
Log on to the ApsaraDB for SelectDB console.
In the upper-left corner of the page, select the region where the instance is located.
On the Instances page, click the target Instance ID to go to the Instance Details page.
In the left-side navigation pane, click Monitoring and Alerts.
On the Monitoring and Alerts page, click the Alert Management tab, and then click Create SelectDB Monitoring Alert.
NoteAfter you click the Alert Management tab, the SelectDB Monitoring Alert List page may take 3 to 5 seconds to load.
On the Create SelectDB Monitoring Alert page, configure the parameters.
You can create an alert rule using either a static threshold or a custom PromQL query.
Static threshold: Provides preset alert metrics. You can select a metric to quickly create an alert rule by defining a condition.
Custom PromQL: Create an alert rule by writing a custom PromQL query. This is useful for metrics that are not in the preset list.
Static threshold
Parameter
Description
Example
Alert Rule Name
The name of the alert rule.
CPU utilization alert
Check Type
Select Static threshold.
static threshold
Instance
Select the instance for the alert rule.
The default value is Traverse, which means the alert rule applies to all instances.
selectdb-cn-7213n****
Cluster
Select the cluster for the alert rule.
The default value is Traverse, which means the alert rule applies to all clusters.
selectdb-cn-7213n****-be
Alert contact group
Select an alert contact group.
The supported alert contact groups vary based on the Prometheus instance type. The available options change based on the selected Prometheus instance type.
SelectDB monitoring alert
Alert metric
Select the metric for the alert. Each alert contact group provides a different set of metrics.
CPU utilization
Alert condition
Define the condition that triggers an alert based on the selected metric.
An alert is sent when the CPU utilization is greater than 80%.
Filter Conditions
No filter
None
Data Preview
The Data Preview section displays the PromQL query corresponding to the alert condition and charts the metric's values over time.
By default, the chart shows the real-time value for a single resource. Use the filters in this section to view data for different resources and time ranges.
NoteThe alert threshold is displayed as a red dashed line in the chart. The curve turns red when the threshold is met and remains blue otherwise.
Hover over the time-series curve to view resource details at a specific point in time.
Select a time range on the time series chart to view the curve for that period.
None
Duration
Generate alert event when condition is met: Triggers an alert if any data point meets the threshold.
Generate alert event when condition persists for N minutes: Triggers an alert only if the condition persists for at least N minutes.
1
Alert Level
The severity level of the alert. The default level is Default. Severity increases in the order: Default, P4, P3, P2, and P1.
P2
Alert Content
The content of the alert notification that users receive. You can use Go template syntax to customize parameter variables in the alert message.
Node: {{$labels.pod_name}} CPU utilization {{$labels.metrics_params_opt_label_value}} {{$labels.metrics_params_value}}%, current value {{ printf "%.2f" $value }}%
Alert Notification
Simple Mode: Lets you quickly configure the Notification Contacts, Notification Period, and Whether to Resend Notifications.
Standard Mode: Lets you select a notification policy. If no policy exists, click Create Notification Policy to create one. For more information, see Notification policy.
ImportantQuickly specifying a notification policy ensures that alert events from the current alert rule are matched by the selected notification policy. However, these events may also be matched by other notification policies that use fuzzy matching. The relationship between alert events and notification policies is many-to-many.
None
Advanced Settings
Alert Check Cycle: The interval at which the alert rule is evaluated. The default and minimum value is 1 minute.
Check After Data Is Complete: Ensures data is complete before evaluation. This is enabled by default, and we recommend keeping this setting.
Tags: Assign tags to the alert. Notification policies can use these tags for matching.
Annotations: Set annotations for the alert.
Alert check cycle: 1 minute
Check After Data Is Complete: Yes
Tags (labels): None
Annotations: None
Custom PromQL
Parameter
Description
Example
Alert Rule Name
The name of the alert rule.
Pod CPU utilization greater than 80%
Check Type
Set to Custom PromQL.
Custom PromQL
Instance
Select the instance for the alert rule.
selectdb-cn-7213n****
Cluster
Select the cluster for the alert rule.
selectdb-cn-7213n****-be
Reference Alert Contact Group
Select an alert contact group.
The supported alert contact groups vary based on the Prometheus instance type. The available options change based on the selected Prometheus instance type.
SelectDB monitoring alert
Reference Metrics
(Optional) Select a reference metric to use its PromQL template as a starting point. You can then modify the pre-filled query.
The available reference metrics are automatically filtered based on the selected Prometheus instance type.
NoteThe template provided by the Reference Metrics is incomplete. You must modify it to create a valid Custom PromQL Statements.
99th Percentile Query Latency
Custom PromQL Statements
Use a PromQL query to define the alert expression.
avg(doris_fe_query_latency_ms{quantile="0.99",pod=~,cluster_id=~}) by (cluster_id) > 300
Data Preview
The Data Preview section displays the PromQL query corresponding to the alert condition and charts the metric's values over time.
By default, the chart shows the real-time value for a single resource. Use the filters in this section to view data for different resources and time ranges.
NoteThe alert threshold is displayed as a red dashed line in the chart. The curve turns red when the threshold is met and remains blue otherwise.
Hover over the time-series curve to view resource details at a specific point in time.
Select a time range on the chart to zoom in on the time-series curve for that period.
None
Duration
If the alert condition is met, an alert event is generated.: Triggers an alert if any data point meets the threshold.
Generate alert event when condition persists for N minutes: Triggers an alert only if the condition persists for at least N minutes.
1
Alert Level
The severity level of the alert. The default level is Default. Severity increases in the order: Default, P4, P3, P2, and P1.
Default
Alert Content
The content of the alert notification that users receive. You can use Go template syntax to customize parameter variables in the alert message.
Namespace: {{$labels.namespace}}/Pod: {{$labels.pod_name}}/Disk device: {{$labels.device}} utilization exceeds 90%, current value {{ printf "%.2f" $value }}%
Alert Notification
Simple Mode: Lets you quickly configure the Notification Contacts, Notification Period, and Whether to Resend Notifications.
Standard Mode: Lets you select a notification policy. If no policy exists, click Create Notification Policy to create one. For more information, see Notification policy.
ImportantQuickly specifying a notification policy ensures that alert events from the current alert rule are matched by the selected notification policy. However, these events may also be matched by other notification policies that use fuzzy matching. The relationship between alert events and notification policies is many-to-many.
None
Advanced Settings
Alert Check Cycle: The interval at which the alert rule is evaluated. The default and minimum value is 1 minute.
Check After Data Is Complete: Ensures data is complete before evaluation. This is enabled by default, and we recommend keeping this setting.
Tags: Assign tags to the alert. Notification policies can use these tags for matching.
Annotations: Set annotations for the alert.
Alert check cycle: 1 minute
Check After Data Is Complete: Yes
Tags (labels): None
Annotations: None
After you configure the parameters, click Save. The alert rule becomes active immediately.
Recommended alert configurations
You can configure alerts based on your requirements. The following table lists common metrics and provides recommended settings.
Metric | Threshold | Duration (min) | Notes |
Query rate | >5000 | 5 | The average query latency in milliseconds (ms). Adjust the threshold based on your business requirements. We recommend that you configure this alert. |
99th percentile query latency | >60000 | 5 | The long-tail query latency in milliseconds (ms). Adjust the threshold based on your business requirements. We recommend that you configure this alert. |
ClusterQuerySuccessRate | <90 | 5 | The success rate of SQL queries. We recommend that you configure this alert. |
CPU Utilization | >80 | 15 | The CPU utilization of the BE cluster. This is a common operational metric. We recommend that you configure this alert. |
Memory Usage | >80 | 15 | The memory utilization of the BE cluster. This is a common operational metric. We recommend that you configure this alert. |
FE CPU usage | >60 | 15 | The CPU utilization of the FE cluster. We recommend that you configure this alert. If the resource is insufficient, submit a ticket to request a free scale-out. |
FE JVM memory utilization | >80 | 15 | The JVM memory utilization of the FE cluster. We recommend that you configure this alert. If the resource is insufficient, submit a ticket to request a free scale-out. |
ClusterFailNodeCount | >0 | 1 | The number of restarts for underlying nodes in the compute cluster. Configure as needed. |
Base compaction score | <1500 | 15 | A higher value indicates greater data compaction pressure on compute nodes. We recommend that you configure this alert. |
Cumulative compaction score | <1500 | 15 | A higher value indicates greater data compaction pressure on compute nodes. We recommend that you configure this alert. |
Cache hit rate | <90 | 15 | The cache hit ratio affects query latency. We recommend that you configure this alert. If this metric drops, evaluate whether to scale out the cluster. For more information, see Scale clusters. |
User Connection Count | >150 | 15 | The total number of connections from database users to the database. The maximum number of connections per user is 200 by default. This metric is specific to your business workload, so we recommend configuring this alert based on your typical usage. |
Queries per second (QPS) | None | None | This metric is workload-dependent. Configure as needed. |
Disk write IOPS | None | None | An underlying system metric, typically not a primary business concern. Configure as needed for advanced troubleshooting. |
Disk read IOPS | None | None | An underlying system metric, typically not a primary business concern. Configure as needed for advanced troubleshooting. |
InstanceOssStorageUsage | None | None | Configure this alert if you need to monitor storage usage. |
ClusterDataLoadRate | None | None | Configure this alert if you need to monitor data import speed. |
ClusterCacheIOWx | None | None | An underlying system metric, typically not a primary business concern. Configure as needed for advanced troubleshooting. |
ClusterCacheIORx | None | None | An underlying system metric, typically not a primary business concern. Configure as needed for advanced troubleshooting. |
PodNetworkFlowRx | None | None | An underlying system metric, typically not a primary business concern. Configure as needed for advanced troubleshooting. |
PodNetworkFlowTx | None | None | An underlying system metric, typically not a primary business concern. Configure as needed for advanced troubleshooting. |
ClusterRemoteStorageIORx | None | None | An underlying system metric, typically not a primary business concern. Configure as needed for advanced troubleshooting. |