Configure alert rules

Updated at:

ApsaraDB for SelectDB integrates the alerting functionality of Alibaba Cloud Application Real-Time Monitoring Service (ARMS). In the SelectDB console, you can configure alert rules. You can configure monitoring items. When an alert rule for a monitoring item is triggered, all contacts in the alert contact group are notified. You can also maintain alert contact groups for monitoring items to ensure that relevant contacts are promptly notified when an alert is triggered. By configuring alert rules for important monitoring metrics, you can be promptly notified when the metric data of a SelectDB instance becomes abnormal, allowing you to quickly identify and troubleshoot faults. This topic describes how to configure alert rules in the SelectDB console.

Note

You can also configure alert rules for ApsaraDB for SelectDB instances in the CloudMonitor console. For more information, see Cloud service monitoring.

Prerequisites

  • An ApsaraDB for SelectDB service-linked role, AliyunServiceRoleForSelectDB, must be created. By default, this role has permission to access the ARMS service. For more information, see Service-linked role.

  • If you want to report monitoring data from your ApsaraDB for SelectDB instance to your own ARMS for centralized monitoring and alerting, you must first activate the ARMS service. For more information, see Activate ARMS.

Procedure

  1. Log on to the ApsaraDB for SelectDB console.

  2. In the upper-left corner of the page, select the region where the instance is located.

  3. On the Instances page, click the target Instance ID to go to the Instance Details page.

  4. In the left-side navigation pane, click Monitoring and Alerts.

  5. On the Monitoring and Alerts page, click the Alert Management tab, and then click Create SelectDB Monitoring Alert.

    Note

    After you click the Alert Management tab, the SelectDB Monitoring Alert List page may take 3 to 5 seconds to load.

  6. On the Create SelectDB Monitoring Alert page, configure the parameters.

    You can create an alert rule using either a static threshold or a custom PromQL query.

    • Static threshold: Provides preset alert metrics. You can select a metric to quickly create an alert rule by defining a condition.

    • Custom PromQL: Create an alert rule by writing a custom PromQL query. This is useful for metrics that are not in the preset list.

    Static threshold

    Parameter

    Description

    Example

    Alert Rule Name

    The name of the alert rule.

    CPU utilization alert

    Check Type

    Select Static threshold.

    static threshold

    Instance

    Select the instance for the alert rule.

    The default value is Traverse, which means the alert rule applies to all instances.

    selectdb-cn-7213n****

    Cluster

    Select the cluster for the alert rule.

    The default value is Traverse, which means the alert rule applies to all clusters.

    selectdb-cn-7213n****-be

    Alert contact group

    Select an alert contact group.

    The supported alert contact groups vary based on the Prometheus instance type. The available options change based on the selected Prometheus instance type.

    SelectDB monitoring alert

    Alert metric

    Select the metric for the alert. Each alert contact group provides a different set of metrics.

    CPU utilization

    Alert condition

    Define the condition that triggers an alert based on the selected metric.

    An alert is sent when the CPU utilization is greater than 80%.

    Filter Conditions

    No filter

    None

    Data Preview

    The Data Preview section displays the PromQL query corresponding to the alert condition and charts the metric's values over time.

    By default, the chart shows the real-time value for a single resource. Use the filters in this section to view data for different resources and time ranges.

    Note
    • The alert threshold is displayed as a red dashed line in the chart. The curve turns red when the threshold is met and remains blue otherwise.

    • Hover over the time-series curve to view resource details at a specific point in time.

    • Select a time range on the time series chart to view the curve for that period.

    None

    Duration

    • Generate alert event when condition is met: Triggers an alert if any data point meets the threshold.

    • Generate alert event when condition persists for N minutes: Triggers an alert only if the condition persists for at least N minutes.

    1

    Alert Level

    The severity level of the alert. The default level is Default. Severity increases in the order: Default, P4, P3, P2, and P1.

    P2

    Alert Content

    The content of the alert notification that users receive. You can use Go template syntax to customize parameter variables in the alert message.

    Node: {{$labels.pod_name}} CPU utilization {{$labels.metrics_params_opt_label_value}} {{$labels.metrics_params_value}}%, current value {{ printf "%.2f" $value }}%

    Alert Notification

    • Simple Mode: Lets you quickly configure the Notification Contacts, Notification Period, and Whether to Resend Notifications.

    • Standard Mode: Lets you select a notification policy. If no policy exists, click Create Notification Policy to create one. For more information, see Notification policy.

      Important

      Quickly specifying a notification policy ensures that alert events from the current alert rule are matched by the selected notification policy. However, these events may also be matched by other notification policies that use fuzzy matching. The relationship between alert events and notification policies is many-to-many.

    None

    Advanced Settings

    • Alert Check Cycle: The interval at which the alert rule is evaluated. The default and minimum value is 1 minute.

    • Check After Data Is Complete: Ensures data is complete before evaluation. This is enabled by default, and we recommend keeping this setting.

    • Tags: Assign tags to the alert. Notification policies can use these tags for matching.

    • Annotations: Set annotations for the alert.

    • Alert check cycle: 1 minute

    • Check After Data Is Complete: Yes

    • Tags (labels): None

    • Annotations: None

    Custom PromQL

    Parameter

    Description

    Example

    Alert Rule Name

    The name of the alert rule.

    Pod CPU utilization greater than 80%

    Check Type

    Set to Custom PromQL.

    Custom PromQL

    Instance

    Select the instance for the alert rule.

    selectdb-cn-7213n****

    Cluster

    Select the cluster for the alert rule.

    selectdb-cn-7213n****-be

    Reference Alert Contact Group

    Select an alert contact group.

    The supported alert contact groups vary based on the Prometheus instance type. The available options change based on the selected Prometheus instance type.

    SelectDB monitoring alert

    Reference Metrics

    (Optional) Select a reference metric to use its PromQL template as a starting point. You can then modify the pre-filled query.

    The available reference metrics are automatically filtered based on the selected Prometheus instance type.

    Note

    The template provided by the Reference Metrics is incomplete. You must modify it to create a valid Custom PromQL Statements.

    99th Percentile Query Latency

    Custom PromQL Statements

    Use a PromQL query to define the alert expression.

    avg(doris_fe_query_latency_ms{quantile="0.99",pod=~,cluster_id=~}) by (cluster_id) > 300

    Data Preview

    The Data Preview section displays the PromQL query corresponding to the alert condition and charts the metric's values over time.

    By default, the chart shows the real-time value for a single resource. Use the filters in this section to view data for different resources and time ranges.

    Note
    • The alert threshold is displayed as a red dashed line in the chart. The curve turns red when the threshold is met and remains blue otherwise.

    • Hover over the time-series curve to view resource details at a specific point in time.

    • Select a time range on the chart to zoom in on the time-series curve for that period.

    None

    Duration

    • If the alert condition is met, an alert event is generated.: Triggers an alert if any data point meets the threshold.

    • Generate alert event when condition persists for N minutes: Triggers an alert only if the condition persists for at least N minutes.

    1

    Alert Level

    The severity level of the alert. The default level is Default. Severity increases in the order: Default, P4, P3, P2, and P1.

    Default

    Alert Content

    The content of the alert notification that users receive. You can use Go template syntax to customize parameter variables in the alert message.

    Namespace: {{$labels.namespace}}/Pod: {{$labels.pod_name}}/Disk device: {{$labels.device}} utilization exceeds 90%, current value {{ printf "%.2f" $value }}%

    Alert Notification

    • Simple Mode: Lets you quickly configure the Notification Contacts, Notification Period, and Whether to Resend Notifications.

    • Standard Mode: Lets you select a notification policy. If no policy exists, click Create Notification Policy to create one. For more information, see Notification policy.

      Important

      Quickly specifying a notification policy ensures that alert events from the current alert rule are matched by the selected notification policy. However, these events may also be matched by other notification policies that use fuzzy matching. The relationship between alert events and notification policies is many-to-many.

    None

    Advanced Settings

    • Alert Check Cycle: The interval at which the alert rule is evaluated. The default and minimum value is 1 minute.

    • Check After Data Is Complete: Ensures data is complete before evaluation. This is enabled by default, and we recommend keeping this setting.

    • Tags: Assign tags to the alert. Notification policies can use these tags for matching.

    • Annotations: Set annotations for the alert.

    • Alert check cycle: 1 minute

    • Check After Data Is Complete: Yes

    • Tags (labels): None

    • Annotations: None

  7. After you configure the parameters, click Save. The alert rule becomes active immediately.

Recommended alert configurations

You can configure alerts based on your requirements. The following table lists common metrics and provides recommended settings.

Metric

Threshold

Duration (min)

Notes

Query rate

>5000

5

The average query latency in milliseconds (ms). Adjust the threshold based on your business requirements. We recommend that you configure this alert.

99th percentile query latency

>60000

5

The long-tail query latency in milliseconds (ms). Adjust the threshold based on your business requirements. We recommend that you configure this alert.

ClusterQuerySuccessRate

<90

5

The success rate of SQL queries. We recommend that you configure this alert.

CPU Utilization

>80

15

The CPU utilization of the BE cluster. This is a common operational metric. We recommend that you configure this alert.

Memory Usage

>80

15

The memory utilization of the BE cluster. This is a common operational metric. We recommend that you configure this alert.

FE CPU usage

>60

15

The CPU utilization of the FE cluster. We recommend that you configure this alert. If the resource is insufficient, submit a ticket to request a free scale-out.

FE JVM memory utilization

>80

15

The JVM memory utilization of the FE cluster. We recommend that you configure this alert. If the resource is insufficient, submit a ticket to request a free scale-out.

ClusterFailNodeCount

>0

1

The number of restarts for underlying nodes in the compute cluster. Configure as needed.

Base compaction score

<1500

15

A higher value indicates greater data compaction pressure on compute nodes. We recommend that you configure this alert.

Cumulative compaction score

<1500

15

A higher value indicates greater data compaction pressure on compute nodes. We recommend that you configure this alert.

Cache hit rate

<90

15

The cache hit ratio affects query latency. We recommend that you configure this alert. If this metric drops, evaluate whether to scale out the cluster. For more information, see Scale clusters.

User Connection Count

>150

15

The total number of connections from database users to the database. The maximum number of connections per user is 200 by default. This metric is specific to your business workload, so we recommend configuring this alert based on your typical usage.

Queries per second (QPS)

None

None

This metric is workload-dependent. Configure as needed.

Disk write IOPS

None

None

An underlying system metric, typically not a primary business concern. Configure as needed for advanced troubleshooting.

Disk read IOPS

None

None

An underlying system metric, typically not a primary business concern. Configure as needed for advanced troubleshooting.

InstanceOssStorageUsage

None

None

Configure this alert if you need to monitor storage usage.

ClusterDataLoadRate

None

None

Configure this alert if you need to monitor data import speed.

ClusterCacheIOWx

None

None

An underlying system metric, typically not a primary business concern. Configure as needed for advanced troubleshooting.

ClusterCacheIORx

None

None

An underlying system metric, typically not a primary business concern. Configure as needed for advanced troubleshooting.

PodNetworkFlowRx

None

None

An underlying system metric, typically not a primary business concern. Configure as needed for advanced troubleshooting.

PodNetworkFlowTx

None

None

An underlying system metric, typically not a primary business concern. Configure as needed for advanced troubleshooting.

ClusterRemoteStorageIORx

None

None

An underlying system metric, typically not a primary business concern. Configure as needed for advanced troubleshooting.