Enable recovery notifications to receive alerts when a monitored object returns to normal.
For example, create an alerting rule to monitor host CPU utilization. The rule triggers an alert when utilization exceeds 95% and sends a recovery notification when it drops to 95% or below. The following shows a sample configuration based on Create an alerting rule.
In the Add Annotation section, set the title to CPU utilization for ${host} spikes and the description to CPU utilization for ${host} reaches ${value}.
-
Query and Statistics:
* | select promql_query_range('cpu_util') from metrics limit 1000Retrieves CPU utilization data.
-
Group Evaluation: Automatic tagging
Automatically groups time series query results.
-
Trigger Condition: Data match, value > 95, Severity: High
Triggers a high-severity alert when value exceeds 95.
-
Add Annotation: Set annotations such as title and description, using field variables like ${host}. Add tags and annotations.
-
Recovery Notification: Turn on the Recovery Notification switch.
A recovery notification is an alert with the status Resolved instead of Triggered. When enabled, Log Service sends a recovery notification if the previous check triggered an alert but the current check no longer meets the trigger condition.
Recovery notifications appear as alerts with the alert status set to Resolved.