A scheduled task is a task that runs automatically based on its schedule settings. On the scheduled task page in Operation Center, you can view the list of scheduled tasks in your workspace, inspect their configurations in the production environment, and perform management operations.
Background
In DataWorks, a scheduled task is a task that runs automatically based on its schedule. You can view all scheduled tasks on the scheduled task management page. These tasks can be run automatically on a schedule or manually on demand. To run a task manually, you can use the backfill data and test features to generate a backfill instance or a test instance for specific data processing or debugging. For an automatically scheduled task, the system creates and runs a new scheduled instance when its trigger conditions are met. For example, at 22:00 every night, the system generates the scheduled instances for the following day.
Changes made to a scheduled task in the production environment typically affect instances that are generated after the change is made. Most modifications to a scheduled task take effect at the next scheduled run time because the scheduled instance for the current day is generated at 22:00 on the previous day. In contrast, a data backfill instance is generated immediately based on the current task configuration and runs as soon as it is manually triggered. For detailed O&M operations such as creating, deploying, running, pausing scheduling, and undeploying tasks, see Basic O&M operations for a scheduled task.
Limitation
Only users of DataWorks Professional Edition or a more advanced edition can use the node aggregation, upstream analysis, and downstream analysis features in the DAG view. For more information about upgrading your edition, see Select an edition and billing method.
Notes
-
Do not operate on the
workspace-name_rootnode. This is the root node of the workspace. All scheduled instances depend on this node. If you freeze this node, scheduled instances cannot run. -
All changes to a task must be deployed to take effect in the production environment. A deployment may fail, be blocked, or result in an unexpected version. Therefore, after you deploy a task, go to the page to verify the changes:
-
Use the DAG to check whether the upstream and downstream dependencies of the task are correct.
-
Use node details to check whether the parameter configurations of the task in the production environment are as expected.
-
Go to the scheduled task page
-
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Operation Center.
-
After you go to Operation Center, in the left-side navigation pane, click . On the Auto Triggered Node page, view the scheduled task list or DAG.
View scheduled tasks
The scheduled task list displays tasks that have been submitted to the scheduling system. You can use Smart Search or Filter to locate the tasks you want to view.
Smart search mode
With the smart search feature for scheduled tasks, you only need to enter search content, and the system automatically parses it and quickly filters out the task list you want to view.
-
Perform a smart search.
Click the Smart Search button in the scheduled task filter bar. In the dialog that appears, enter what you want to search for, such as
Sort by task type, and then press Enter to confirm. The system automatically matches and displays the relevant tasks. -
Save a new view.
If you want to reuse the search criteria from a smart search later, click in the search bar. In the Save View dialog, specify a custom View Name, and then click Save. You can then find and use the new view in the view search bar for future task searches.
NoteIf you no longer want to use the new view for task searches, find the view name you created in the view search bar. Hover over the view name and click the ... button on the right, and then select Modify or Delete the view.
-
Close smart search.
If you need to perform a precise search by using filter conditions, press Esc or click the Close Smart Search button in the task toolbar to exit the smart search mode.
Filter mode
With the filter feature for scheduled tasks, you can use the provided filter conditions to precisely filter out the task list you need.
-
Simple filtering.
In the toolbar, you can select multiple filter conditions one by one based on Task Name/Task ID, Task type, Expired Within Last 180 Days, Referenced, and other criteria to filter out the tasks you want to view.
-
Advanced filtering.
Click the Filter button in the scheduled task filter bar. You can combine multiple conditions such as Task Name/Task ID, Compute Engine Type, Baseline, and Priority to precisely locate the tasks you need.
-
Paused (frozen) nodes: A frozen task is not automatically scheduled and blocks its downstream tasks from running. Specifically, a paused (frozen) scheduled task generates frozen instances, and frozen instances are not scheduled to run and block their downstream tasks from running.
-
Orphan nodes: An orphan node is a task that does not depend on any parent node. This type of task is not automatically scheduled. If an orphan node has many downstream dependencies, it affects the running of downstream tasks. For more information, see Handle orphan nodes.
-
Expired nodes: Nodes that are not within their validity period do not generate instances and do not run.
Manage scheduled tasks
Manage individual tasks
To manage a specific scheduled task, find the task in the scheduled task list and use the corresponding features in the Actions column. The following table describes the available features:
|
Feature |
Description |
|
|
DAG |
The DAG displays the upstream and downstream dependencies of a scheduled task. You can perform related operations on tasks in the DAG. For more information, see View the DAG of a scheduled task. |
|
|
Test |
Used to test whether a scheduled task runs as expected. After you perform a test, a test instance is generated. You can view the running status of the test task on the Test Instance page. For more information, see Test instances. Note
Test runs also generate business data. Proceed with caution. |
|
|
Run |
Used to backfill historical data or data for future time periods. After you perform a backfill data operation, a backfill instance is generated. You can view the running status of the backfill data task on the Data Backfill Instance page. For more information, see Backfill data instances. |
|
|
More |
Freeze |
When a task and its downstream tasks do not need to run for a period of time, you can pause (freeze) the task. A frozen scheduled task generates frozen instances. Frozen instances are not executed and block the execution of their downstream tasks. Note
Do not operate on the |
|
Unfreeze |
Used to unfreeze a frozen scheduled task. |
|
|
Change Owner |
Used to change the owner of the current task. |
|
|
Modify Scheduling Resource Group |
Used to change the resource group for scheduling used by the scheduled task. |
|
|
Modify label |
You can modify, add, or disassociate tags for the scheduled task asset. To manage tags centrally, go to Tag management. |
|
|
Configure Data Quality Rules |
Used to configure data quality monitoring rules for the current task. When the task runs, data is validated against the rules. For more information, see Data Quality overview. |
|
|
View Instances |
Click this option to go to the Scheduled Instance page that shows the instances generated by the current task. |
|
|
View Lineage |
Used to view the lineage information of the current task. |
|
|
View cycle task details |
Used to view the basic properties of the task. For more information about the related parameters, see View node details. |
|
|
Add Alert Rule |
Used to configure custom alert rules to help you monitor the daily running status of scheduled tasks. After the configuration is complete, you can view the monitoring rules added on the page. For more information, see Configure monitoring rules. |
|
|
Add to Baseline Monitoring |
Used to add the current task to a baseline. For more information about baselines, see Intelligent baseline overview. |
|
Batch task operations
To perform batch operations on scheduled tasks, select the desired tasks in the scheduled task list and use the options below the list, such as Freeze and Unfreeze.
View the DAG of a scheduled task
Click the DAG button in the Actions column of a scheduled task to go to the DAG details page of the task.
DAG panel features
On the scheduled task DAG details page, you can use the following features to aggregate nodes, analyze downstream tasks, and adjust the DAG display style in the DAG panel.
|
Feature |
Description |
|
|
|
You can click the feature icons in the upper-left corner as needed to aggregate node information by the following dimensions:
|
|
|
|
When there are too many scheduled tasks or too many levels, you can use the Downstream Analysis feature to count the number of downstream tasks affected by the current task. |
|
|
|
You can click the feature icons in the upper-right corner as needed to adjust the DAG display style. |
|
DAG operations
On the scheduled task DAG details page, right-click a node in the workflow to view its upstream and downstream dependencies, code details, and other related information. The available operations are as follows:
-
Show Ancestor Nodes: Expands the upstream tasks of the current task by level. A maximum of 6 levels can be expanded. You can use this feature to identify which tasks affect the data output of the current task.
-
Show Descendent Nodes: Expands the downstream tasks of the current task by level. A maximum of 6 levels can be expanded. You can use this feature to identify which tasks are affected by the data output of the current task.
-
View Code: Used to check the code details of the current task in the production environment. If the code is not as expected, verify whether the latest version of the task has been deployed successfully. For information about deploying a production task, see Deploy tasks.
-
Edit Node: Click this option to go to the Data Studio page and open the current task.
For other operations, see Manage scheduled tasks.
View node details
You can view task properties and expand the details to view detailed attributes, context parameters, operation logs, and code to verify whether the scheduling parameters, resource groups, and code in the production environment are as expected. The procedure is as follows:
On the scheduled task DAG details page, click a node in the workflow. In the node information that appears, click expand details to view the related information on the tabs below.
|
Feature |
Description |
|
Property |
You can view the schedule settings of the task in the production environment. For more information about the parameters, see Schedule settings. Note
If the parameter configurations in the Data Studio page and the scheduled task page of Operation Center are inconsistent, verify whether the task changes have been deployed successfully.
|
|
Operation Log |
Used to view the operation records of the task, including the operation time, operator, and specific operations. |
|
Code |
Used to view the latest code of the current task in the production environment. If the code is not as expected, verify whether the latest version of the task has been deployed successfully. For more information, see Deploy tasks. |