Interval detection

Updated at:

A single static threshold is often ineffective for fluctuating metrics like RT and QPS because they require different alert thresholds at different times of day. When a data point suddenly falls outside the upper or lower boundary, the system generates an anomaly detection event. This method is best suited for monitoring metrics with stable trends.

Use cases

  • Monitor golden metrics such as response time and request rate. If a service's response time suddenly exceeds its normal range, the system sends an immediate alert, helping administrators quickly locate and resolve the issue.

  • Monitor the CPU and memory usage of your servers. If a server's resource utilization consistently exceeds the predicted threshold, the system automatically generates an anomaly event. This helps your team promptly adjust resource allocation and prevent potential system crashes.

  • Monitor key metrics such as query rate and concurrent connection count. When a thread's metrics fall outside the normal range, the system automatically triggers an anomaly event, helping you promptly optimize application performance.

  • In a microservice architecture, interactions and dependencies between services are complex. This feature lets teams monitor the resource usage and response performance of each service. When a service behaves abnormally, teams can quickly pinpoint the issue and ensure the stability of the entire system.

Example:

For a website, traffic below 1,000 during the day (for example, 10:00 to 18:00) is abnormal, but traffic that exceeds 1,000 at night (for example, 22:00 to 06:00) might indicate an attack. In this scenario, the normal baseline of the metric changes over time. If you configure a static threshold to trigger an alert when the value is below 1,000, you will receive alerts for abnormal traffic during the day but will not be notified of a potential attack at night. By using dynamic thresholds, the system can intelligently identify the normal baseline and automatically update the threshold range.

Prerequisites

Before you begin, make sure that you have:

  • An application that reports data to Managed Service for OpenTelemetry. For details, see Integration guide

Configure interval detection

  1. Log on to the Managed Service for OpenTelemetry console.

  2. In the left-side navigation pane, choose Alert Management > alert rule.

  3. On the alert rule page, click Create alert rule.

  4. On the Create alert rule page, enter a custom Alert Name, and set Alert Detection Type to interval detection.

  5. In the Alert Object section, select the alert application, metric type, and filter conditions based on your requirements.

    Parameter

    Description

    Alert Application

    The application to monitor. Interval detection supports configuring an alert for only a single application.

    Metric Type

    The metric type to monitor. For more information, see Metrics for alert rules.

    After you select a metric type, the system automatically calculates the upper and lower boundaries and renders them in real time. You can preview the metric trend in the Alert Condition section.

    Note
    • The condition fields for the alert rule and the Filter Condition options vary based on the selected Metric Type.

    • The initial rendering takes 2 to 4 seconds.

    • For information about how the upper and lower boundaries are calculated, see How the threshold range is calculated.

    Filter Condition

    Filters the metric data to narrow the monitoring scope.

    Dimensions for the alert metric:

    • Traverse: The alert content includes the specific dimension value that triggered the alert.

    • None: The alert content includes the sum of all values for this dimension.

    • = : The alert content includes only data for the specified dimension value.

    • !=: The alert content includes data for all dimension values except the one specified.

    • Contain: The alert content includes only data for dimension values that contain the specified string.

    • Do Not Contain: The alert content includes data for all dimension values that do not contain the specified string.

    • Regular Expression Match: The alert content includes only data for dimension values that match the specified regular expression.

    In the alert rule section, set Alert Trigger Mode to Single Condition. Configure the alert condition to trigger if the Average of call count within a specified time range is Outside dynamic threshold boundaries. Set the alert level to P1 : critical. You can adjust the threshold sensitivity by using the tolerance slider and click alert quantity prediction to preview when alerts would have been triggered.

  6. In the alert rule section, configure the Alert Condition.

    Parameter

    Description

    Alert Trigger Mode

    Interval detection only supports single-condition triggers.

    Alert Condition

    Configure the specific alert condition, which includes the following factors:

    • Last X minutes: The time range for monitoring. The maximum value is 60 minutes.

    • Metric Measure: The quantifiable data or value of the metric. The available options vary based on the metric type, such as call count or response time.

    • Aggregation Method: The calculation performed on the metric data, such as average, maximum, or minimum. The available methods depend on the metric and its measure.

    • Comparison Method: The comparison method used to identify anomalous data points. Interval detection supports the following three comparison methods:

      • Outside dynamic threshold boundaries: Triggers an alert if a data point falls outside the automatically calculated boundaries.

      • Outside dynamic threshold upper boundary: Triggers an alert if a data point is above the automatically calculated upper boundary.

      • Outside dynamic threshold lower boundary: Triggers an alert if a data point falls below the automatically calculated lower boundary.

    • Alert Level: The severity level of the alert, from P1 to P4.

    In the data preview area, the blue line represents actual data points, and the green area represents the range between the upper and lower boundaries.

    Tolerance

    Expands or shrinks the boundaries based on the automatically calculated range. A higher tolerance (sliding to the right) widens the boundaries, making it less likely for data to be considered an anomaly and trigger an alert. A lower tolerance (sliding to the left) narrows the boundaries, which makes it more likely for data to be diagnosed as an anomaly and trigger an alert.

    Alert quantity prediction

    Displays the estimated number of times the metric will exceed the threshold within the selected time period. You can click a specific alert count to view the historical metric values that would have triggered an alert.

    We recommend that you use the alert quantity prediction feature every time you create or modify an alert rule. This feature analyzes historical data to predict the number of alerts for a selected time period, which helps you adjust the threshold. For more information, see Alert quantity prediction.

  7. Configure Alert Notification and Advanced Alert Settings.

    Parameter

    Description

    Alert Notification

    Simple Mode

    • Notification Objects: For information about how to create notification objects, see Notification objects.

    • Notification Period: Select the time period during which alert notifications are sent.

    • Repeat Policy:

      • If an escalation policy is not required, a notification is sent only once while the alert is active.

      • Set a repeat frequency. If an alert is not resolved, notifications are repeatedly sent at the specified frequency until the alert is resolved.

    Standard Mode

    Notification Policy:

    • Do not specify notification policy: No notification is sent when an alert is triggered. Notifications are sent only when the matching rules of a notification policy are met.

    • Specify a notification policy: When an alert is triggered, Application Real-Time Monitoring Service (ARMS) sends notifications based on the specified notification policy. You can select an existing notification policy or create one. For more information, see Notification policy.

    Advanced Alert Settings

    No Data

    This setting is used to handle missing data or data from composite metrics and period-over-period comparisons. When the alert metric does not meet the specified conditions, you can configure the system to report the alert data as 0 or 1, or to not trigger an alert.

    For more information, see Alert management terms.

  8. Click Save.

How the threshold range is calculated

The dynamic thresholds feature in ARMS primarily uses the Prophet algorithm. After you configure a dynamic thresholds task, ARMS runs in the background once every 24 hours to learn from the metric's historical data from the past 7 days. It extracts features such as trends and seasonality to generate a prediction curve for the next 24 hours. Based on the metric's own fluctuations (error variance), it estimates a range for the metric's data for the next day. When you configure a dynamic thresholds alert, you can preview the upper and lower boundaries calculated by the algorithm.

Unlike static threshold recommendations, you do not need to manually update the alert rule when business changes cause the metric's baseline to shift. This is because ARMS continuously learns the metric's characteristics once a day and predicts boundaries for only the next day. Therefore, you do not need to manually adjust the threshold multiple times.

Alert quantity prediction

The alert quantity prediction feature analyzes historical data to forecast the number of alerts that would have been triggered within a selected period and shows the exact time each alert would have occurred.

How it works

ARMS analyzes the last 24 hours of metric data to predict how many alerts your proposed threshold would have generated. ARMS also provides details showing the exact times the metric value would have exceeded the threshold. You can use this information to adjust thresholds to better fit your business needs.