Scheduling dependencies in DataWorks define the upstream-downstream relationships between auto-triggered nodes (task nodes that run periodically in the scheduling system). After you configure scheduling dependencies, the system ensures that downstream node instances are triggered only after all upstream node instances run successfully, guaranteeing that data is produced and consumed in the correct order. This topic describes the basic concepts, dependency types, and configuration methods of scheduling dependencies, helping you gain an overall understanding before configuration and quickly find the documentation that applies to your scenario.
Overview
Scheduling dependency is a mechanism in DataWorks that defines upstream-downstream relationships between nodes. By configuring scheduling dependencies, you can specify that a node starts running only after its specified upstream nodes run successfully, ensuring the correct order of data processing. After dependencies are configured, the DataWorks scheduling system automatically orchestrates the execution order: a downstream instance is triggered only when all upstream instances run successfully and conditions such as time and resource availability are met.
DataWorks establishes dependency relationships between nodes by matching node output names with node input names. From the perspective of the current node, configuring dependencies involves two core operations:
Configure upstream dependencies (node inputs)
Add inputs to the current node to specify its upstream dependencies. In the configuration panel, you can establish dependency relationships by searching for an upstream node by its node output name (recommended), node name, or node ID. The current node instance starts running only after all specified upstream node instances run successfully.Configure downstream outlets (node outputs)
Configure output names for the current node as unique identifiers that downstream nodes can use to depend on it. We recommend that you use theproject_name.table_nameformat (for example,my_project.dim_user) as the output name to clearly indicate the data table produced by the node. After configuration, downstream nodes can depend on the current node by referencing this output name.
Auto-parse (optional): For SQL-type nodes, DataWorks can automatically parse theINSERTandSELECTstatements in the code, identify input and output tables, and automatically generate dependency configurations. You can also manually adjust the auto-parsed results. For the node types that support auto-parsing, see Support for the automatic parsing feature.
Each node must have at least one output name. The system automatically generates a default output for each node . Even if you delete all custom outputs, this default output is retained.
Rules and constraints
Effective upon deployment: Scheduling dependency configurations take effect only after the node is deployed to Operation Center. Configurations made during development are not automatically synchronized to the scheduling environment.
Upstream and downstream scheduling status: Dependency relationships take effect only when instances of both the upstream and downstream nodes are generated and in normal scheduling status. If a node is improperly configured or an upstream instance is abnormal, the node may become isolated and cannot be scheduled.
Circular dependency restriction: The system prohibits circular dependencies between nodes (for example, A depends on B and B depends on A), including both direct and indirect cycles. If a circular dependency is detected during deployment, the system blocks the deployment and returns an error.
Dependency types
DataWorks provides two major categories of scheduling dependencies: same-cycle dependencies and cross-cycle dependencies, which apply to different business scenarios. Same-cycle dependencies associate with the nearest upstream instance in the same cycle by default. After you enable advanced scheduling dependency configurations, you can further select Specified Range or Specified Set to precisely control the range of dependent upstream instances, covering flexible scenarios such as cross-time-zone and cross-window dependencies.
Prerequisite concepts
A cycle is a relative concept whose meaning is defined by the scheduling time of a node. A scheduling cycle refers to the time offset between two adjacent scheduling instances of a node, which is determined by its scheduling frequency. For example, for a daily scheduling task, the previous cycle corresponds to the instance from the previous day; for an hourly scheduling task, it corresponds to the instance from the previous hour.
Scheduling frequency | One cycle |
Daily, weekly, monthly, and yearly scheduling | 1 day Note For weekly, monthly, and yearly scheduling tasks, instances are still generated on a daily basis (instances on non-scheduling days are dry-run instances). Therefore, dependency calculation is based on the day granularity, and the previous-cycle instance may be in dry-run status. |
Hourly scheduling | An interval at the hour level |
Minute scheduling | An interval at the minute level (for example, every 5 minutes) |
Dependency types
DataWorks provides the following dependency types based on how dependencies are mounted:
Nearest dependency (same-cycle dependency): The downstream instance is mounted to the nearest upstream instance in the same cycle based on the proximity principle.
Specified range: You specify a continuous range by using start and end offsets to precisely control the range of dependent upstream instances. This is applicable to scenarios such as cross-time-zone data dependencies.
Specified set: You select multiple discrete upstream instances. This is applicable to scenarios where the downstream node depends on multiple specific upstream cycle instances.
Cross-cycle dependency: You can quickly specify the previous-cycle instance result of a given node. The specified node can be the current node itself (self-dependency), a first-level child node of the downstream, or any other node.
Specified range and Specified set are available only after you turn on the Enable advanced scheduling dependency configuration switch (disabled by default) in the scheduling settings of DataStudio. The upper limit of the range is the previous day and the current day: for minute-level tasks, the maximum offset is ±1440 minutes; for hourly tasks, the maximum offset is ±24 hours; for daily and higher-frequency tasks, the maximum range is from 00:00 of the previous day to 23:59 of the current day.
Comparison of the four dependency types
Example: A daily scheduling node A produces the dim_user table, and downstream node B consumes this table:
Nearest dependency (same-cycle dependency)
The downstream daily task depends on the data produced by the upstream daily task on the same day. For example, today's sales report (node B) must wait until today's sales total (node A) has been computed.Specified range
The downstream task depends on all upstream instances within a specific time window. For example, the daily task in the China region (node B) depends on all 24 instances of the hourly task in the India region (node A) within the[-3, 21]hour window.Specified set
The downstream task depends only on upstream instances at specific points in time. For example, a summary task (node B) only needs to wait for the upstream data collection task (node A) to complete instances generated at0:00,6:00,12:00, and18:00.Cross-cycle dependency (depends on the previous cycle)
The downstream task depends on the complete data produced by the upstream task in the previous cycle, or achieves serial execution through self-dependency. For example, a T+1 report (node B) depends on data produced by node A yesterday, or an hourly task depends on its own instance from the previous hour to avoid concurrency.
Comparison item | Same-cycle dependency | Cross-cycle dependency (depends on the previous cycle) | ||
Nearest dependency (same-cycle dependency) | Specified range | Specified set | ||
Description | The current instance of this node depends on the execution result of the upstream node's instance in the same cycle. The nearest upstream instance is mounted based on the proximity principle. | Uses start/end offsets to specify a continuous range, allowing precise control over the range of upstream instances that are depended on. | Selects multiple discrete upstream instances by specifying them individually. | The current instance of this node depends on the execution result of a specified node's instance from the previous cycle. The specified node can be this node itself (self-dependency), a first-level child node of the downstream, or any other node. |
Representation in the DAG | Displayed as a solid line. | Displayed as a solid line. | Displayed as a solid line. | Displayed as a dashed line. |
Typical scenario | Node B needs to read data produced by node A today. | Cross-timezone data dependency (for example, a daily task in China depends on a continuous range of hourly task instances in India or Saudi Arabia). | Depends on only some upstream cycle instances (for example, only the 0:00, 6:00, 12:00, and 18:00 time points). | A node depends on data produced yesterday (for example, T-1 data retrieval). Hourly or minute-level tasks achieve serial execution through self-dependency to avoid concurrent execution of multiple cycle instances. |
Configuration method | Supports automatic parsing, workflow line drawing, and manual addition. | Configure start/end offsets. | Select the desired discrete set from the upstream task instances. | In the "Previous Cycle" section of the scheduling configuration panel, select the dependency type and specify the node ID. |
Whether advanced configuration is required | No | Yes | Yes | No |
Note: Same-cycle dependencies and cross-cycle dependencies can coexist between the same pair of nodes, but their respective business purposes must be clearly defined. If you only need a cross-cycle dependency, remember to delete the same-cycle dependency that was automatically generated by the system. Otherwise, the downstream instance still needs to wait for the upstream instance in the current cycle to complete before it can run, resulting in unexpected delays.
Scheduling dependency configuration guide
To ensure the integrity and maintainability of the scheduling chain, all nodes must have upstream dependencies configured before they can be deployed to Operation Center for automatic scheduling. If a node has no data dependency, it must depend on a virtual node or the root node. When configuring scheduling dependencies, you need to analyze the business logic of the node, identify the dependency targets and dependency types, and select the most appropriate configuration method to build a data workflow that is both robust and clearly structured.
1. Identify dependency targets
Before configuring dependencies, complete the following preparations:
Analyze lineage: Confirm whether the tables or partitions produced by the upstream match the tables or partitions read by the downstream.
Check scheduling properties: Make sure that the scheduling cycle, effective time, scheduling parameters, and other properties of the node are correctly configured, because scheduling properties directly affect dependency mounting behavior.
Select dependency objects based on how the current node depends on data.
Scenario 1: Depend on the direct output of an upstream node |
|
Scenario 2: Depend on non-scheduled upstream data (data-readiness driven) |
|
Scenario 3: No direct data dependency, but a business logic association exists |
|
2. Select a dependency type
If the current node depends on the direct output of an upstream node (Scenario 1), you need to further confirm whether the dependent data is the output of the upstream node from the same scheduling cycle or from a cross-cycle.
Core decision criteria
Determine which cycle's output data the downstream node actually reads from the upstream node. In most scenarios, a node periodically writes data to a specific partition of a table by using scheduling parameters for dynamic resolution. You can refer to Scheduling parameter sources and expressions to understand how scheduling parameters are replaced. If you need to depend on a node in the same workspace, you can check the scheduling parameter configuration of that node.
How to confirm
Nodes in the same workspace: Check the scheduling parameters in the upstream node code. Determine whether the partition written after parameter replacement is "today's" partition or "yesterday's" partition.
In the development environment, check the scheduling parameter configuration and code details of the upstream node. In the production environment, check the parameter replacement results in the instance details.
Nodes in a different workspace: Use Data Map to view the partition information and change history of the upstream table.
Confirm the partition values that are actually written each day.
Select a type
The downstream code reads the partition of the current day or current cycle from the upstream node: Same-cycle dependency.
The downstream code reads the partition of the previous day or previous cycle from the upstream node: Cross-cycle dependency.
Hourly or minutely tasks that must run strictly in the scheduled order of instances: Cross-cycle dependency, which means depending on the current node itself.
The downstream node aggregates upstream data across time zones by business date (for example, a daily task in China aggregates local hourly tasks from regions such as India and Saudi Arabia): Enable advanced configuration and use a specified range to cover the instances of the corresponding local business day.
The time window of the downstream dependency crosses a natural day boundary (for example, an early-morning batch task processes data only from the previous afternoon to the early morning of the current day): Enable advanced configuration and use a specified range to define the cross-day continuous window.
The downstream node only needs to depend on the most recent few cycle instances of the upstream node, instead of waiting for all upstream instances of the current day by default: Enable advanced configuration and use a specified range to limit the dependency scope.
The downstream node depends on only a few discrete time points of the upstream node (for example, 0:00, 6:00, 12:00, and 18:00): Enable advanced configuration and use a specified set to select the corresponding instances.
Consequences of failing to correctly confirm lineage:
Missing dependency risk: If a table lineage exists but no scheduling dependency is configured, the downstream task starts before the upstream instance succeeds, resulting in no data being read or incomplete data.
Parameter mismatch risk: If a dependency is configured but the partition parameters are misaligned (for example, the upstream node produces today's partition but the downstream node reads yesterday's partition), data logic errors and quality anomalies will occur.
3. Configure dependencies
Based on the dependency objects and dependency types confirmed in steps 1 and 2, select an appropriate configuration method to configure dependencies.
DataWorks allows tasks with different scheduling frequencies to depend on each other. By combining same-cycle/cross-cycle dependencies with scheduling parameters, you can implement a wide range of scheduling scenarios. For more information, see:
To precisely control the range of upstream instances for dependencies (such as specifying a continuous offset range or selecting discrete instances), enable the "Enable advanced scheduling dependency configuration" toggle in DataStudio > Scheduling Settings, and configure dependencies by using the Specified range or Specified set method.
4. Verify scheduling dependencies
After the configuration is complete and before you deploy the node, you must perform the following verification:
Verification method | Description |
When you commit a node, use this method to verify whether the dependency changes of the current node version are as expected and to assess the impact of the changes on the production environment. When auto-parsing is enabled, to ensure normal data generation in the production environment, you must confirm the scheduling changes of the node when you commit it. You can use this feature to ensure that dependency changes do not affect data generation by production tasks. | |
After a node is deployed, use this method to verify whether the dependencies of the production scheduling task in Operation Center are as expected.
|
Impact of removing dependencies on downstream tasks
During task O&M or iteration, you may need to remove or adjust existing scheduling dependencies.
Before you remove a dependency, make sure that you assess the impact on the scheduling behavior of downstream tasks to avoid orphaned downstream tasks or data incidents. For more information about orphaned nodes, see Orphaned nodes.
Downstream dependency scenario | Impact after dependency removal | Risk level |
The downstream task depends only on the current node. | The downstream task becomes an orphaned node, loses the upstream triggering mechanism, and is no longer automatically scheduled. | High |
The downstream task depends on multiple parent nodes. | The downstream task may start before the upstream data is ready, which causes data missing or calculation errors. | Medium |
The downstream task depends on cross-cycle instances. | If the cross-cycle dependency is removed, the downstream task may read data of an incorrect business date, which causes data logic errors. | Medium |
Use cases
Offline data warehouse layered construction: Configure full-link dependencies across ODS → DWD → DWS → ADS to ensure that layered data is generated in order.
Standard ETL pipeline: Configure same-cycle dependencies to ensure that downstream tasks are strictly executed only after upstream instances succeed, which guarantees the order and consistency of the data processing pipeline.
Next-day (T+1) reports: Configure cross-cycle dependencies (offset -1) so that today's task depends on the complete business data from the previous day, enabling accurate next-day data analysis and output.
Multi-cycle mixed aggregation: Configure cross-cycle dependencies so that a daily task depends on all cycle instances of an hourly task, ensuring that underlying data is fully ready before aggregation.
External data readiness trigger: Configure custom dependencies or check nodes to confirm that external files have arrived or interfaces are ready before triggering the workflow, enabling cross-system scheduling coordination.
Complex workflow control: Use virtual nodes to aggregate multi-branch dependencies as workflow control milestones, simplifying the dependency chain structure and improving monitoring visibility.
Cross-timezone multi-region data aggregation: A data warehouse in China processes data from regions worldwide. By specifying an interval to mount upstream hourly instances corresponding to the local business day of each region, cross-timezone offsets are covered (for example, India [-3, 21], Saudi Arabia [-5, 19]).
Cross-day window aggregation: A downstream task aggregates data within a non-calendar-day time window. By specifying an interval, you can flexibly select a continuous window between the previous day and the current day (for example, [-12, 4] covers from 12:00 of the previous day to 04:00 of the current day).
Depend on only the most recent upstream window: A downstream task needs only the most recent instances from the upstream task (for example, the last 6 hours). By specifying an interval, you can limit the mount range and avoid waiting for all upstream instances of the current day by default.
Depend on discrete upstream time points: A downstream task only needs the output of the upstream task at specific hours (for example, 00:00, 06:00, 12:00, and 18:00). You can specify a set to select the corresponding instances.
FAQ
The following section describes typical scenarios. For more FAQ about scheduling dependencies, see FAQ about dependencies.
Node uniqueness.
A node has different forms in the development and production environments but remains unique: The scheduling dependency configurations of the same node can differ between the development and production environments. This means the same node can have two different forms in the development and production environments, but the node itself is unique.
You must remove downstream dependencies in both the development and production environments before you take a node offline: Due to node uniqueness, to ensure that downstream tasks can correctly retrieve data and run, DataWorks requires you to first remove the dependency in the downstream node scheduling configuration, then reconfigure the upstream nodes that the downstream node needs to depend on, and commit and deploy the changes. You can take the upstream task offline only after the dependency is removed in both the development and production environments.
Instance generation method.
When you create a node, make sure that the upstream and downstream nodes use the same instance generation method. If their instance generation method: Immediately after deployment settings differ, the upstream node may generate instances on the current day while the downstream node generates instances the next day, causing downstream instances to become Scenario: Isolated nodes.
If you change the scheduling cycle of an existing node and select the option to generate instances immediately after deployment, the previously generated instances are not automatically deleted when you modify the scheduling dependencies. The dependencies of cycle instances generated on the day of deployment may be inconsistent. For more information, see Instance generation method: Immediately after deployment.
An error indicating that the number of upstream dependencies exceeds 200 occurs when you use OpenAPI to update a task.
Error details: 'One file could not have more than 200 inputs 'One file could not have more than 200 inputs'.
You can add virtual nodes between upstream and downstream nodes in DataStudio to reduce the number of direct upstream dependencies of the current node. For more information about virtual node configuration, see Zero load node.