Why is a threshold-triggered alert rule triggered with a delay?

Before you begin, understand the following two concepts:

• Period of a metric

  The period of a metric is the interval between two consecutive data points and the aggregation window. For example, if the period is 60 seconds, one data point is generated every 60 seconds. Each value is either the last reported value or an aggregate of all values within that 60-second window.

• Alert detection period

  The detection period of an alert rule is determined by the metric period. If a multi-metric expression contains multiple periods, the shortest period prevails. For example, with a 60-second detection period, CloudMonitor queries the latest metric data every 60 seconds.

If the metric period in an alert rule is long (for example, 300 or 900 seconds), writing the metric data to disk may be delayed, or the data point may arrive slightly after the detection time. As a result, the alert may be triggered with a delay of one detection period. If the metric period is short, no significant delay occurs.

Availability monitoring alerts, group process monitoring alerts, metric-based alerts of cloud services, site monitoring alerts, and PromQL-based alerts for Hybrid Cloud Monitoring all use fixed-threshold expressions. This applies to single metrics, multiple metrics, complex expressions, and period-based comparisons. The following two issues use simple single-metric expressions as examples:

Issue 1: The metric data is reported periodically. The data meets the alert condition for N consecutive times, but an alert is triggered with a delay of nearly one period.

Cause

The detection time is slightly earlier than the data point timestamp. As a result, the alert is triggered with a delay of nearly one period.

Example

Assume the metric period is 300 seconds and the detection period is also 300 seconds. The alert condition requires the average metric value to exceed the threshold for three consecutive periods. The following table lists the data points.

Assume the first detection occurs at 10:06:00, which is slightly earlier than the data point at 10:06:30. The metric data queried at 10:06:00 is still value 5 from the previous period, which does not exceed the threshold.

• At the second detection at 10:11:00, the queried value is 15, which exceeds the threshold for the first time.

• At the third detection at 10:16:00, the queried value is 14, which exceeds the threshold for the second time.

• At the fourth detection at 10:21:00, the queried value is 13, which exceeds the threshold for the third time. An alert notification is sent.

Based on the data above, you might expect the third threshold breach to occur at 10:16:30. However, the alert is triggered with a delay of 4.5 minutes, which is close to one period.

Issue 2: The metric data is reported sparsely and aperiodically. An alert is triggered with a delay of one period.

Cause

If metric data is reported aperiodically or sparsely, writing the data to disk is delayed.

Example

• Example 1: Assume the metric period is 300 or 900 seconds. If data is reported sparsely or aperiodically, the data point may be delayed by up to 5 minutes. For example, querying the 10:00:00 data point at 10:01:00 returns no result, but querying the same data point at 10:05:00 returns a result. Therefore, the latest data may be unavailable at detection time, and the alert is triggered with a delay of one period.

• Example 2: The alert condition requires the metric value to exceed 10 at least once. The following table shows the alert detection process.