Configure monitoring and alerting

Job event alerts

Subscribe to system event alerts for jobs by configuring conditions. Batch configuration of event alerts is supported.

1. Log on to the Cloud Monitor console.

2. In the navigation pane on the left, choose Event Center > Event Subscription.

3. On the Subscription Policy tab, click Create Subscription Policy.

4. On the Create Subscription Policy page, configure the parameters. For more information about the parameters, see Manage event subscriptions (recommended).

   • Subscription Type: System Event.

   • Product: Select Realtime Compute for Apache Flink.

   • Event Name: Supports Job Run Failed (not supported when using the ARMS monitoring service), Post-mortem Processing for ECS Downtime, and Impact of Proactive O&M on ECS events.

   • Event Content: You can specify jobs or configure batch alerts by entering the following Flink information in the event content.

     • Workspace ID: Configure event alerts for all jobs in all namespaces under the target workspace. To view the workspace ID, see How do I view information such as the workspace ID?.

     • Namespace Name: Configure event alerts for all jobs in the target namespace.

     • Deployment Job Name: Configure event alerts for specific jobs. Separate multiple job names with a comma (,). If there are jobs with the same name under your account, use the DeploymentID instead.

     • DeploymentID: Configure event alerts for specific jobs. Separate multiple IDs with a comma (,). You can find the ID on the Deployment Details tab of the Flink job page as the deployment ID.

   Note

   If you do not set Application Group, Event Content, or Event Resource, the subscription applies to all workspaces within your account.

Workflow event alerts

Subscribe to system event alerts for Flink workflows by configuring conditions. Batch configuration of event alerts is supported. For more information about workflows, see Manage workflows.

1. Get the resource ID of the workflow node.

   1. Log on to the Cloud Monitor console.

   2. In the navigation pane on the left, choose Event Center > System Event.

   3. On the Event Monitoring tab, set Product to Realtime Compute for Apache Flink and Event Name to Workflow Task Status Change, and then click Search.

   4. Find the resource ID of the workflow node below.

      On the Event Monitoring page, select Realtime Compute for Apache Flink from the Product drop-down list. In the event list, find the record with the event name flink:Workflow:TaskStateChange Workflow Task Status Change. You can view the resource ID of the workflow node in the Resource column.

      The resource format is acs:flink:cn-hangzhou:<AlibabaCloudAccountId>:resourceId/workspaceId/<workspaceId-namespaceId>#workflowDefinitionName/<workflowDefinitionName>#taskDefinitionName/<taskDefinitionName>. You can also use this format to directly construct the resource ID for your workflow node.

      Parameter	Description
      <AlibabaCloudAccountIdOfWorkspace>	The Alibaba Cloud account ID that owns the Flink workspace.
      <workspaceId-namespaceId>	A combination of the workspaceId and namespaceId, joined by a hyphen (-). workspaceId: The workspace ID. For more information, see Workspace management and operations. namespaceId: The namespace name.
      <workflowDefinitionName>	The workflow name.
      <taskDefinitionName>	The workflow node name.

      Note

      There is usually a delay of several minutes before workflow status change events appear in CloudMonitor.

2. Subscribe to event notifications.

   1. In the navigation pane on the left, choose Event Center > Event Subscription.

   2. On the Subscription Policy tab, click Create Subscription Policy.

   3. On the Create Subscription Policy page, set the parameters for the subscription policy. For more information about the parameters, see Manage event subscriptions (recommended).

      • Subscription Type: System Event.

      • Product: Select Realtime Compute for Apache Flink.

      • Event Name: Select Workflow Task Status Change.

      • Event Content: Supports entering parameters such as toState: FAILED (workflow failed), toState: SUCCESS (workflow succeeded), and fromState: SCHEDULED, toState: RUNNING (workflow state changed from scheduled to running).

      • Event Resource: Enter the workflow resource ID obtained in Step 1. Separate multiple IDs with a comma (,).

      • Event Type, Event Level, Application Group: Do not set any of these.

Configure a single job (Management Console)

You can create a new alert rule for a target job, or you can create an alert rule template and then use it to create alerts for target jobs. This can speed up the configuration of monitoring and alerting.

1. Go to the alert configuration page.

   1. Log on to the Management Console and click Console in the Actions column for the target workspace.

   2. On the O&M > Deployments page, click the name of the target job.

   3. Click the Alert Configuration tab.

2. On the Alert Rules tab, choose Add Alert Rule > Custom Rule.

   You can also choose Add Alert Rule > Rule Template. By adding and using an alert template, you can create alert rules directly or with minor modifications, which speeds up the configuration process.

3. Enter the alert rule information.

   Category	Parameter	Description
   Rule Details	Rule Name	Must start with a letter and can only contain lowercase letters, numbers, and underscores (_). The length is limited to 3 to 64 characters.
   	Description	A note about the rule.
   	Content	Configure the conditions that trigger an alert. The system compares the specified metric value with the threshold at a specified interval. If the result meets the condition, an alert is automatically triggered. Metric: Restart Count in 1 Minute: The number of times the Job Manager has restarted in 1 minute. Checkpoint Count in 5 Minutes: The number of successful checkpoints in 5 minutes. Emit Delay: The business latency, which is the time difference between when data is generated and when it leaves the Source operator. The unit is seconds. Important The data generation time depends on the timestamp recorded in the external system. If the external system does not have a timestamp, or if the timestamp is written incorrectly, the Emit Delay value will be inaccurate and will not reflect the true latency. Configure alerts with multiple metrics to determine the real event. For more information, see Recommended configuration examples and templates for monitoring and alerting. IN RPS: The number of input records per second. The unit is records/second. OUT RPS: The number of output records per second. The unit is records/second. Source Idle Time: The time that the source has not processed data. The unit is milliseconds. Job Failed: The job has failed. Time Difference: The length of the historical data time window that the system queries backward for each check. The unit is minutes. Operator: Supports >= and <=. Threshold: The value to compare with the metric. If you select the >= operator, the system uses the MAX value from the vertical axis. If the maximum value within the time difference is >= the threshold, the alert rule is triggered. If you select the <= operator, the system uses the MIN value from the vertical axis. If the minimum value within the time difference is <= the threshold, the alert rule is triggered. For example, assume you are monitoring the "Checkpoint Count in 5 Minutes" metric. The time difference is set to 10 minutes, the threshold is 2, and the operator is "<=". Every minute, the system checks the historical data from the last 10 minutes. If it finds any 5-minute interval where the number of successful checkpoints was less than or equal to 2, it triggers an alert.
   	Effective Period	The time when the alert monitoring is active. You can specify it to be active only during the day (9:00 to 18:00). By default, it is active all day.
   	Alert Frequency	Send an alert only once within a continuous number of minutes. Supports 1 minute to 1440 minutes (24 hours).
   Notification Method	Notification Method	You can select multiple notification methods. The supported methods are: DingTalk Email SMS Webhook Phone Ensure that the recipient's phone number has been verified. Otherwise, this method will not work. You can click Notification Contact Management below. If the Phone column for the target contact on the contact tab shows Not verified, click to complete the verification. Important Ensure that you have created and added available notification contacts. Otherwise, the alert notification will fail. For example, if you select DingTalk as the notification method, select DingTalk and add a DingTalk notification contact of the DingTalk chatbot type. For more information about the billing of alert management, see Billing of alerts.
   	Notification Contact	You can notify multiple contacts at the same time. You can select or search for contacts. Before selecting a contact, you must click Notification Contact Management on the right to create one. For more information, see Workspace management and operations.
   	Alarm Noise Reduction	Click Advanced Settings, then you can turn on the Alarm Noise Reduction switch. After you turn on alarm noise reduction, alerts will not be sent for scenarios where the job can recover quickly, such as short-term failovers triggered by cluster scheduling or automatic tuning. Alerts are sent only when the threshold conditions you set are met continuously.
   	No Data Alarms	Click Advanced Settings, then you can turn on the No Data Alarms switch and enter the continuous no-data time information. After you turn on this feature, the system monitors for scenarios where no monitoring data is reported. If no data is reported within the selected time period, an alert is triggered. No monitoring data is usually reported when the JobManager is abnormal, the job stops unexpectedly, or the reporting link is abnormal.

4. Click OK.

   The saved alert rule is enabled by default and appears in the alert rule list. You can choose to stop, edit, or delete it.

Configure single or multiple jobs (ARMS console)

1. Log on to the Management Console.

2. In the Actions column for the target workspace, choose More > Monitoring Metric Configuration to go to the ARMS console.

   The workspace name, workspace ID, and corresponding Prometheus instance name are displayed at the top.

   The navigation pane on the left includes Dashboard List, Service Discovery, Metric Management, Alert Rules, and Settings. The agent status is displayed in the upper right. On the Metrics tab of the Service Discovery page, you can filter metrics by job name, metric type, and other conditions. The metric list displays the metric name, job name, metric type, and other information.

3. Click Alert Rules on the left to create an alert rule.

   • Check Type: Supports metric alerts through static thresholds and custom PromQL (excluding alert metrics already supported by Flink).

   • Filter Conditions: Batch alert configuration is supported. For Namespace, enter the namespace name. Selecting all indicates all namespaces in the workspace. For Deployment Job, enter the deployment ID of the target job in the namespace (found on the Deployment Details tab of the Flink job). Selecting all indicates all jobs in the namespace.

   For more information about configuration parameters, see Create a Prometheus alert rule. You can also create a Prometheus alert rule template. For more information, see Create a Prometheus alert rule template.