How to configure automated O&M rules in the DataWorks Operation Center-DataWorks(DataWorks)-阿里云帮助中心

Configure automated O&M rules based on your emergency response experience. When trigger conditions are met, the system automatically performs operations such as terminating running instances or rerunning failed nodes, improving service stability and reducing night-time maintenance.

Background

In DataWorks, the automated O&M feature consists of automated termination of running node instances and automated rerun.

Automated termination of running node instances

If a node that runs on an exclusive resource group for scheduling triggers a custom alert rule about resource groups, the system uses the specified automated O&M rule to terminate specific instances generated for the node. For example, if the resource utilization rate of an exclusive resource group for scheduling reaches 80% and persists for 10 minutes, the system automatically terminates the execution of non-auto triggered node instances with priorities 1 and 3 on the exclusive resource group for scheduling.
Automated rerun

A node is automatically rerun based on the automated rerun rule in the following scenarios: 1. The status of the node is Failed and the automatic rerun property is not configured for the node. 2. The node fails because the node running times out.

Limitations

Limits on permissions: Only Alibaba Cloud accounts, RAM users to which the AliyunDataWorksFullAccess policy is attached, and workspace administrators can manage automated O&M rules.
Limits on resource groups:
- Automated O&M rules about automated termination of running node instances take effect only for nodes that are run on an exclusive resource group for scheduling take effect for nodes that are run on and for which an alert rule about the resource utilization rate of an exclusive resource group for scheduling is configured.
- Automated O&M rules about creating automated rerun rules take effect only for nodes that are run on a serverless resource group take effect for nodes that are run on.
Limits on features:
- You can associate multiple automated O&M rules about automated termination of running node instances with the same alert rule.
- You can create only one automated O&M rule for creating automated rerun rules in each workspace.
- You can view the execution records that are generated for automated O&M rules within the previous 30 days.

Go to the Automatic page

Log on to the DataWorks console. In the target region, click Data Development and O&M > Operation Center in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Operation Center.
In the left navigation bar, select O&M Assistant > Automatic O&M to go to the Automated O&M page.

Create a rule

On the Automatic O&M > Rule Management page, you can create Create Terminate Running Instance and Auto Rerun automated O&M rules. Each rule requires trigger conditions, filter conditions to target specific nodes, an optional blacklist to exclude nodes, and constraint rules to limit when the rule takes effect.

Create a rule for terminating running instances

In Automatic O&M, configure rules to automatically terminate instances that meet custom rules. Supported instance types: recurring instances, data backfill instances, test instances, one-time task instances, and manually triggered workflow instances. The main configuration parameters are as follows:

Section	Parameter	Description
Trigger Condition	Associated Monitoring Rule	The alert rule that you want to associate with the automated O&M rule. If the alert rule is triggered, the node instance is automatically terminated. Note For information about how to create a monitoring rule. You can associate only monitoring rules whose Object Type is Scheduling Resource Group and whose Trigger Condition is Resource Group Usage.
Filter Conditions	Workspace	The name of the workspace to which the automated O&M rule is applied.
	Instance Type	The type of the node instance to which the automated O&M rule is applied.
	Scheduling Cycle	Select the scheduling cycles to which the O&M rule applies. If you set Instance Type to Scheduled Instance or Data Backfill Instance, you must configure this parameter.
	Priority	The priority of the node instances to which the automated O&M rule is applied. A larger value indicates a higher priority.
	Status	The status of the node instances to which the automated O&M rule is applied.
Blocklist		The nodes that meet the conditions specified in the automated O&M rule but on which you do not want to perform the O&M operation. To add a node to the blacklist, enter the name or ID of the node in the search box.
Constraints on Rule	Effective Period	The time range within which the rule is effective. The O&M operation is performed only when the rule conditions are met and the rule is triggered within this period. Outside this period, the operation is not performed even if conditions are met.
	Maximum Effective Times	The maximum number of times the rule can be triggered and executed. Note Each time before an automated O&M rule is executed, the system checks whether the trigger condition is met. If the trigger condition is not met, the automated O&M rule is not executed.
	Minimum Effective Interval	The minimum interval at which the automated O&M rule can be triggered.

Create an automated rerun rule

In Automatic O&M, configure the Auto Rerun feature for nodes that meet the specified Trigger Condition. Supported instance types: periodically scheduled instances, data backfill instances, smoke testing instances, one-time task instances, and manually triggered workflow instances.

When the instance is a recurring instance, automatic rerun only checks instances with a data timestamp of yesterday.

For example, if the current date is June 5, 2025, only recurring instances with a data timestamp of June 4, 2025 will be automatically rerun after meeting the automatic rerun trigger conditions.
When the instance is a data backfill instance, test instance, one-time task instance, or manually triggered workflow instance, automatic rerun checks instances created today, yesterday, and the day before yesterday.

For example, if the current date is June 5, 2025, then data backfill instances, test instances, one-time task instances, and manually triggered workflow instances created on June 5, June 4, and June 3 will be automatically rerun after meeting the automatic rerun trigger conditions.

The main configuration parameters for automatic rerun rules are as follows:

Section	Parameter	Description
Trigger Condition	Running status	The specified instance will be automatically rerun when it meets the following conditions and fails to run: The running of a node instance times out. The node instance fails to run but the automatic rerun property is not configured for the node that generates the instance on the Properties tab.
Filter Conditions	Workspace	The name of the workspace to which the automated O&M rule is applied.
	Instance Type	The type of the node instance to which the automated O&M rule is applied.
	Scheduling Cycle	Select the scheduling cycles to which the O&M rule applies. If you set Instance type to Scheduled Instance or Data Backfill Instance, you can configure this parameter.
	Priority	The priority of the node instances to which the automated O&M rule is applied. A larger value indicates a higher priority.
	Logs Contain Keywords	The keyword that you want to identify in the operation logs of the node instance. If the operation logs of the node instance contain the keyword, the automated rerun rule is automatically triggered. The valid values are `abnormal exit` (the node process fails to start or unexpectedly exits) and `out of memory` (the node fails to run and exits due to insufficient memory). Note The automated rerun rule can be triggered for a node whose operation logs contain the `out of memory` keyword only if the node is run on a serverless resource group.
Blacklist	Blocklist	The nodes that meet the conditions specified in the automated O&M rule but on which you do not want to perform the O&M operation. To add a node to the blacklist, enter the name or ID of the node in the search box.
Rerun	Preparation	For computing nodes that run on a serverless resource group, you can select Add CUs for Computing Tasks before the node is rerun. Note Specify the number of CUs added for each rerun to prevent the running of other nodes from being blocked due to competition for resources.
	CUs to Add	In addition to the CUs consumed by the original node instance, add the specified CUs for the rerun instance. The added CUs are used only for the rerun of the instance.
	Rerun Times	The maximum number of times that an automated rerun can be triggered. Valid values: 1 to 10. Unit: times.
	Rerun Interval	The interval between reruns. Valid values: 3 to 30. Unit: minutes.
Constraints on Rule	Effective Period	The time range within which the rule is effective. The O&M operation is performed only when the rule conditions are met and the rule is triggered within this period. Outside this period, the operation is not performed even if conditions are met.

Enable or disable a rule

By default, an automated O&M rule takes effect immediately after the rule is created. To disable the rule, click the icon in the Actions column of the rule.

More operations

Manage rules

To view the information about a rule, go to the Rule Management page and click View in the row of the rule.
To modify a rule, click Edit at the bottom of the View Rule dialog box.
To delete a rule, click Delete in the row of the rule and click Confirm in the dialog box that appears.
To search for a rule, enter its name in the search box on the Rule Management page.

View the execution records of a rule

On the Execution Records page, you can view the execution records of O&M rules, including the execution time, rule owner, and the number of involved nodes. To view the detailed information, click View Details in the row of an execution record.

Note

When an automated O&M rule is triggered, the operation is performed under the identity of the rule owner. You can view the operation in the operation logs of the triggering node instance.

The execution records of an automated O&M rule about automated termination of running node instances include the following information:
- Instances Waiting for Resources/Resource Usage: A line chart that displays the number of instances waiting for resources and the resource usage over time. Hover over the chart to view these metrics for a specific point in time.
- Terminated Node Instances: This section displays all the node instances whose running is terminated.
The execution records of an automated O&M rule about automated rerun of node instances include the following information:
- Automatically rerun instances: A list of instances that were automatically rerun, including their Node Name, Data Timestamp, Instance Type, Task type, and Owner.

Monitor resource groups

After you create an automated O&M rule, the system automatically monitors the resource usage of the resource group specified in the automated O&M rule. For more information about resource group monitoring, see Resource O&M.