Configure node scheduling
Configure scheduling properties for tasks (nodes) in DataWorks, including schedule, dependencies, run policies, and parameters, to ensure tasks run automatically and reliably.
Core concepts
Before configuring scheduling properties, understand the following core concepts:
Tasks and instances: A task (or node) is the business logic code developed in DataWorks. When a task runs based on its Scheduling Settings, one or more running instances are generated. In Operation and Maintenance Center, you manage and monitor running instances of tasks, such as viewing logs and rerunning instances.
Periodic instances: For periodically scheduled tasks, the scheduling system generates corresponding Scheduling Frequency (such as daily or hourly), producing the corresponding periodic instances. For example, an hourly task generates 24 pending hourly instances each day.
Business date: The data processing time of an instance, typically T-1 (yesterday). For example, an instance that runs in the early hours of 2023-01-02 has a business date of 2023-01-01, meaning the instance processes data from January 1. Scheduling parameter replacement is typically based on the business date.
Prerequisites
A node has been created. DataWorks uses nodes for task development. Different engine tasks are encapsulated as different node types. Select the appropriate node for your needs. For details, see Node Development Overview.
The Scheduling Settings page has the Enable Periodic Scheduling switch turned on. After enabling, all tasks in the workspace can run automatically based on their configurations. For details, see Configure workspace scheduling properties.
Configure scheduling properties
In the Scheduling Settings panel on the right side of the node editor page, set various properties to define the automated behavior of the task.
1. Go to the scheduling configuration page
Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the desired workspace and choose in the Actions column.
In Data Studio, double-click the target node to open its editor page.
Click Scheduling Settings on the right side of the node editor page to open the configuration panel.
2. Configure scheduling properties
The scheduling configuration panel includes tabs for Scheduling time, Scheduling Dependency, Scheduling Policy, Scheduling Parameters, Node output parameters, and Associated Role. Configure them as needed.
Scheduling time: Define when a task runs
Define when and how often a task runs automatically.
For nodes within a workflow, the Scheduling time is configured uniformly by the workflow and cannot be modified here. For standalone nodes, configure it here.
Parameter | Description |
Scheduling Frequency | Defines how often a task runs automatically in production. Supports minute, hourly, daily, weekly, monthly, and yearly periods.
|
Effective Date | Set the start and end dates for automatic task scheduling. After the validity period, the task no longer generates new periodic instances. |
Cron expression | This expression is automatically generated based on the time properties configured in the UI. No manual configuration is needed. |
Scheduling time vs. actual run time: The Scheduling time configured here is the expected start time of the task. The actual start time is also affected by upstream task completion, resource group availability and other factors.
Dry run mechanism on non-scheduled days: For weekly, monthly, or yearly tasks, on days not designated for scheduling, the system generates a dry run instance. This instance immediately succeeds and triggers downstream tasks, but does not execute any code or consume compute resources.
Scheduling frequency independence: A task's scheduling frequency is determined solely by its own Scheduling Frequency, independent of upstream tasks. DataWorks supports dependencies between tasks with different scheduling periods (e.g., an hourly task depending on a daily task).
Scheduling dependencies: Define prerequisites for task execution
Define which upstream tasks must complete successfully before the current task can start. Properly configuring dependencies is key to ensuring data processing order and output accuracy.
DataWorks automatically parses and recommends upstream dependencies based on table lineage in code to ensure data integrity. You can also manually add or modify dependencies as needed.
Configure upstream dependencies
In the Scheduling Dependency tab, view the list of upstream dependency nodes parsed from code by the system.
Select the dependency mode for each upstream node:
Same-cycle dependency: The downstream instance today (T) depends on the upstream instance today (T). This is the most common dependency. For example, if daily Task B reads a table produced by daily Task A, Task B should depend on the same cycle of Task A.
Previous-cycle dependency: The downstream instance today (T) depends on the upstream instance yesterday (T-1). For example, today's task needs to read yesterday's aggregated data. For details, see Cross-cycle dependencies.
(Optional) If the system did not automatically parse dependencies, or you need to depend on a non-data-producing node (such as Zero load node), click Add upstream node to configure manually.
Workflow dependencies: Node dependencies within a workflow are managed uniformly by the workflow. This section covers only standalone node dependency configuration. For workflow scheduling configuration, see Configure workflow scheduling properties.
Complex dependency scenarios: DataWorks supports various complex dependency scenarios. Before configuring, review the complex dependency configuration documentation to understand preset dependency rules.
Unsupported dependency scenarios: For data not produced by DataWorks periodic scheduling (such as real-time synced tables, manually uploaded tables, or dimension tables), the system cannot detect data update status, so scheduling dependencies cannot be configured. In such scenarios, depend on the workspace root node or Zero load node for unified scheduling orchestration.
Scheduling policy: Define task run behavior and resources
Define how instances are generated, run behavior (such as timeout and rerun), and required resources.
Parameter | Description |
Instance generation method | Define when periodic instances start generating after a task is deployed.
|
Scheduling Type |
|
Timeout Definition | Set the maximum duration for task execution. After this time, the task is automatically terminated and marked as failed. The default is 3–7 days, with a maximum manual setting of 168 hours (7 days). |
RUN Attribute | Define under which conditions the task can be manually rerun.
|
Auto Rerun upon Failure | When enabled, if a task fails due to transient issues (such as network fluctuations), the system automatically retries to improve scheduling stability.
Note Tasks that fail due to timeout do not trigger automatic reruns. |
Max Parallel Instances | Limits the maximum number of concurrent instances of the same task for concurrency control. The value range is 1–10000. When enabled, excess instances queue until existing instances complete. |
Resource Group | Configure the scheduling resource group for the task. |
Compute Resource/Compute quota | Configure the compute engine resources (such as MaxCompute Quota) for the task. |
Datasets | Mount a dataset (such as Object Storage Service (OSS) or Apsara File Storage (NAS)) to specific node types (such as Shell) so they can be accessed like local files in code. You can configure Mount Path, Advanced Settings (such as read method), and Read Only permissions. |
Parameter configuration: Enable dynamic value passing between tasks
Parameterized configuration makes task code more flexible, enabling dynamic value passing and context transfer between tasks.
Scheduling parameters (input parameters)
If variables are used in the node code (such as ${pt_time}), assign values here. Values can be fixed constants or dynamic expressions using DataWorks built-in parameters (such as $bizdate).
Use Add parameters to manually define, or use Loading parameters in code for automatic identification. In addition, you can use the 
icon to bind parameter values with upstream node output parameters.
The final value of scheduling parameters is determined by the instance's business date and the configured scheduling parameter expressions.
Best practice: After deploying a task, verify the scheduling parameter configuration on the Operation and Maintenance Center > Auto Triggered Task page in production to ensure the configuration meets expectations.
Node output parameters (output parameters)
Define the output of the current node as key-value pairs for downstream nodes to reference. Downstream nodes can consume these values through the Scheduling Parameters configuration via Associate Output Parameter of Ancestor Node.
The Parameter Value can be Constant (fixed strings) or Variable (referencing system or custom input parameters).
Supported node types:
EMR Hive,EMR Spark SQL,ODPS Script,Hologres SQL,AnalyticDB for PostgreSQL, andMySQL.Pass query results: Node output parameters can only pass string values. To pass upstream SQL query result sets downstream, use Assignment node.
Associated role: Securely access other cloud resources
Specify a RAM role for a task to dynamically obtain temporary access credentials through Alibaba Cloud STS (Security Token Service) at runtime, enabling access to other cloud resources (such as OSS) without hardcoding AccessKey pairs, improving security.
Limits
Resource group restriction: Only nodes running on serverless resource groups are supported.
Node type restriction: Only Python, Shell, Notebook, PyODPS 2, PyODPS 3, and PAI DLC nodes are supported.
1. Configure the associated role for a node
On the right side of the node editor page, find and click Run Configuration.
In the scheduling configuration panel, switch to the Associated Role tab.
In the RAM Role drop-down list, select the RAM role you have prepared.
ImportantIf the drop-down list is empty or you cannot find the desired role, see Configure an associated role by using STS to complete the RAM role configuration.
After the configuration is complete, submit the node. This configuration takes effect only for debug runs.
2. Run and verify
PyODPS: When accessing other cloud products (such as OSS), the configured RAM role identity is used. However, when accessing MaxCompute data, the access identity configured for compute resources (project level) is still used automatically.
Configure scheduling properties
After debugging the node, synchronize the Run Configuration from Associated Role to Scheduling Settings in . After deployment, the task runs with the identity of the specified role.
If you configured a custom image in Run Configuration, you must also synchronize the settings to the scheduling configuration.
View the execution role in Operation Center
After the task completes, in Operation and Maintenance Center, view the task instance details to confirm the specified role was used successfully.
Go to .
Find the node instance you ran and click to open its details page.
In the details page, check the Execution Identity field to confirm the task successfully used the specified RAM role.
3. (Optional) View or use temporary credentials in code
After you configure an associated role, the task automatically uses the identity of the RAM role to access other cloud resources at runtime. In most cases, you can run the task without explicitly obtaining credentials in your code.
If you need to explicitly use temporary credentials in your code or you want to view the temporary credentials injected by the system, you can obtain them by using the following methods. DataWorks automatically obtains temporary credentials at runtime and injects them into the runtime environment as environment variables.
PAI DLC nodes do not support obtaining temporary credentials in the ways described in this section.
Method 1: Read environment variables (recommended for Shell and Python)
The system automatically sets the following three environment variables, which you can read directly in your code.
LINKED_ROLE_ACCESS_KEY_ID: The temporary AccessKey ID.LINKED_ROLE_ACCESS_KEY_SECRET: The temporary AccessKey secret.LINKED_ROLE_SECURITY_TOKEN: The temporary security token.
Code example (Python):
This example requires a custom Python image with oss2 installed. For details, see Custom image.
import os
import oss2
# 1. Obtain temporary credentials from environment variables
access_key_id = os.environ.get('LINKED_ROLE_ACCESS_KEY_ID')
access_key_secret = os.environ.get('LINKED_ROLE_ACCESS_KEY_SECRET')
security_token = os.environ.get('LINKED_ROLE_SECURITY_TOKEN')
# Verify credentials obtained
if not all([access_key_id, access_key_secret, security_token]):
raise Exception("Failed to get linked role credentials from environment variables.")
# 2. Initialize the OSS client with temporary credentials
# Assume the role has been granted access to 'your-bucket-name'
auth = oss2.StsAuth(access_key_id, access_key_secret, security_token)
bucket = oss2.Bucket(auth, 'http://oss-<regionID>-internal.aliyuncs.com', 'your-bucket-name')
# 3. Access OSS resources using the client
try:
# List objects in the bucket
for obj in oss2.ObjectIterator(bucket):
print('object name: ' + obj.key)
print("Successfully accessed OSS with linked role.")
except oss2.exceptions.OssError as e:
print(f"Error accessing OSS: {e}")Code example (Shell):
#!/bin/bash
access_key_id=${LINKED_ROLE_ACCESS_KEY_ID}
access_key_secret=${LINKED_ROLE_ACCESS_KEY_SECRET}
security_token=${LINKED_ROLE_SECURITY_TOKEN}
# Access OSS. Replace regionID, bucket_name, and file_name with actual values
echo "ID："$access_key_id
echo "token:"$security_token
ls -al /home/admin/usertools/tools/
# Example: Download a file from OSS to local test_dw.py using ossutil and print its content.
/home/admin/usertools/tools/ossutil64 cp --access-key-id $access_key_id --access-key-secret $access_key_secret --sts-token $security_token --endpoint http://oss-<regionID>-internal.aliyuncs.com oss://<bucket_name>/<file_name> test_dw.py
echo "************************ Success ************************, printing"
cat test_dw.pyMethod 2: Use Credentials Client (recommended for Python)
Code example (Python):
This example requires a custom Python image with oss2 and alibabacloud_credentials installed. For details, see Custom image.
from alibabacloud_credentials.client import Client as CredentialClient
import oss2
# 1. Use the SDK to automatically obtain credentials
# It automatically reads LINKED_ROLE_* credential info from environment variables
cred_client = CredentialClient()
credential = cred_client.get_credential()
access_key_id = credential.get_access_key_id()
access_key_secret = credential.get_access_key_secret()
security_token = credential.get_security_token()
if not all([access_key_id, access_key_secret, security_token]):
raise Exception("Failed to get linked role credentials via SDK.")
# 2. Initialize the OSS client with credentials
auth = oss2.StsAuth(access_key_id, access_key_secret, security_token)
bucket = oss2.Bucket(auth, 'http://oss-cn-hangzhou.aliyuncs.com', 'your-bucket-name')
# 3. Access OSS
print("Listing objects in bucket...")
for obj in oss2.ObjectIterator(bucket):
print(' - ' + obj.key)
print("Successfully accessed OSS with linked role via SDK.")Next steps: Submit and deploy
Scheduling configurations do not take effect immediately after being set in the development environment. The node must be submitted and deployed to the production environment before the scheduling system generates periodic instances and automatically runs tasks based on the latest configuration.
Related documentation
Scheduling parameters: Scheduling parameter format reference
Scheduling time: Scheduling time reference
Scheduling dependencies:
Node output parameters: Node output parameter reference