Same-cycle dependency configuration

Node output

A node output is the output name of the current node. It serves as an identifier (medium) for other nodes to establish dependencies with this node, rather than the actual data produced by the node. Other nodes locate this node through its output name, thereby setting this node as an upstream dependency.

DataWorks automatically generates two output names for each node:

• Random ID: Globally unique and cannot be modified or deleted.

• projectName.nodeName: An output that includes the node name and can be modified. This output name is not automatically updated when the node name changes.

Additionally, DataWorks supports manually adding outputs or using Parse Inputs and Outputs from Code. For information about code parsing support for different node types, see Supported node types for automatic code parsing.

Node input

A node input refers to the upstream nodes that the current node depends on, which can be specified by the upstream node's output name (recommended), node name, or node ID.

The node ID is generated only after the upstream node is deployed to the production environment.

1. Set node dependencies by parsing table lineage from code

Code parsing automatically sets the node's output names and upstream dependencies by analyzing the table lineage in the node's code. After parsing, tables that the node writes to are automatically added as node outputs in the projectname.tablename format, and tables that the node queries are automatically added as node inputs.

For example, when you SELECT a table, it is automatically parsed as an upstream dependency of the current node; when you INSERT into a table, it is automatically parsed as an output of the current node. For the keywords supported by automatic parsing for each node type, see Keywords supported by automatic code parsing for each node type.

• Configure dependencies

  Code parsing supports two methods: manually clicking Parse Inputs and Outputs from Code and Automatic Parsing Before Committing. Both methods work on the same principle. The automatic parsing before submission method automatically parses inputs and outputs when you submit the code and prompts you to set up dependencies.

  For example, in workspace yunwan_lingyi, there is a MaxCompute node mc2 that depends on the output table dws_user_info_all_di of MaxCompute node mc1. The code of node mc2 is as follows:

  INSERT OVERWRITE TABLE ads_user_info_1d PARTITION (dt='${workflow.var}')
  SELECT uid
    , MAX(region)
    , MAX(device)
    , COUNT(0) AS pv
    , MAX(gender)
    , MAX(age_range)
    , MAX(zodiac)
  FROM dws_user_info_all_di
  WHERE dt = '${workflow.var}'
  GROUP BY uid;

  After you click Parse Inputs and Outputs from Code, the system parses the input of the current node as the dws_user_info_all_di table and automatically matches the upstream node's output table name and upstream node name:

  Upstream node output name	Upstream node output table name	Upstream node name	Node ID	Workspace	Owner	Scheduling cycle	Method	Recent run status	Action
  yunwan_lingyi.dws_user_info_all_di	yunwan_lingyi.dws_user_info_all_di	mc1	-	Test Workspace	lingyi01_testcloud_com	Day	Code parsing	No data	Delete

  The system also parses the output of this node as the ads_user_info_1d table. The parsing result is as follows:

  Output name	Output table name	Downstream node name	Owner	Method	Baseline affected by downstream node	Action
  ide.505487297_out	-	-	-	Added by system by default	-	Delete
  ide.mc2	-	-	-	Manually added	-	Delete
  yunwan_lingyi.ads_user_info_1d	yunwan_lingyi.ads_user_info_1d	-	-	Code parsing	-	Delete

  At this point, node mc2 has completed the dependency configuration with node mc1.

• Modify code-parsed dependencies

  If the code-parsed dependencies do not meet expectations, or if there are scenarios that do not support scheduling dependencies (tables with non-periodic data production) that require you to manually delete dependencies, you can refer to the following content to modify the automatically parsed dependencies.

  Operation	Description
  Manually delete parsing results	In the upstream node dependency list, delete the unexpected input, perform the delete operation, and re-parse. After deletion, a corresponding comment is automatically added to the code to prevent the dependency from being added again during the next parsing: --@exclude_input=Remove input --@exclude_output=Remove output
  Manually add inputs and outputs	In the code editor, right-click a table name and select Add Input or Add Output. After the addition, a corresponding comment is automatically added to the code. --@extra_output=Add output --@extra_input=Add input Alternatively, you can add dependencies by referring to manually adding upstream node dependencies through the scheduling configuration panel or setting node dependencies by drawing lines in the workflow panel.

  Important

  DataWorks does not allow you to directly delete a node output that has downstream dependencies. Doing so will cause downstream task execution or data retrieval failures. We recommend that you first adjust downstream tasks by removing the upstream dependency from downstream nodes, and then delete the node output from the upstream node.

• Scenarios excluded from automatic parsing

  Temporary tables in DataWorks (tables in a fixed format defined in workspace table management, such as tables whose names start with t_) are not automatically parsed as outputs of the current node or upstream node dependencies.

• Notes on using automatic parsing

  When using automatic parsing to set up dependencies, make sure that node outputs are unique within the current region. When using automatic parsing in DataWorks development scenarios, note the following:

  • Node creation: Each node has a default node output with the same name as the node. If nodes with the same name exist in the same workspace, you must manually modify the node output of one of the nodes.

  • Code development: Automatic parsing uses the output table of a node as the node output. If two scheduling nodes in the same workspace insert data into the same table, automatic parsing will cause an error in one of the nodes. For more information, see Multiple nodes write data to the same table, and automatic parsing reports that node output names are the same.

  • Dependency configuration: If you use SQL tasks to process the output tables of batch synchronization tasks, to enable SQL tasks to quickly depend on batch synchronization tasks through lineage-based automatic parsing, you must manually configure the output table of the batch synchronization node as a node output, or use the output table name of the batch synchronization task as the node name of the batch synchronization task (the platform automatically creates a node output with the same name as the node). Otherwise, when the downstream SQL node is submitted, the following error may occur: The parent node output name ${projectname.tablename} that the current node depends on does not exist. The current node cannot be submitted. Make sure the parent node with this output name has been submitted.

2. Manually add upstream node dependencies through the scheduling configuration panel

In the Scheduling Settings -> Scheduling Dependency -> Same cycle dependence configuration panel, manually add upstream node dependencies by entering the node output, node name, or node ID of a node. Because node names can be duplicated, we recommend that you use node outputs to establish dependencies.

3. Set node dependencies by drawing lines in the workflow panel

When you set dependencies by drawing lines in the DAG panel of a workflow, DataWorks automatically adds an output in the _out format from the upstream node to the downstream node to establish the node dependency.

1. Nearest dependency (standard same-cycle dependency)

• Meaning: This is the most common and fundamental same-cycle dependency method. It follows the "nearest mounting" principle, where the downstream instance automatically associates with and waits for the latest instance produced in the same business cycle from the upstream node, without additional configuration (there are a few exceptions; for details, see Scheduling configuration principles and examples for complex dependency scenarios).

• Applicable scenarios:

  • Standard daily, weekly, or monthly ETL workflows.

  • One-to-one dependency relationships between upstream and downstream tasks, such as one daily task depending on another daily task.

2. Specified range (depend on a continuous time window)

When a simple one-to-one dependency does not meet your needs—for example, when the downstream task needs to aggregate data from the upstream task over a time period—you can use Specified Range for more flexible configuration.

• Configuration method:

  1. After selecting the dependent node, click Configure in the Actions column and set the dependency type to Specified Range.

  2. Set the left and right offsets relative to the current node's scheduled time. The maximum allowed offset range is -1440 to 1440, in minutes.

• Description: This allows a downstream task to depend on all upstream instances within a continuous time range. The range is a "sliding window" dynamically calculated relative to the downstream task's own scheduled run time, and only instances from the current day and the previous day within the specified range are allowed.

• Core logic example:
  An hourly downstream task depends on another hourly upstream task, with the range configured as [-2h, +2h].
  

  • When the downstream instance at 12:00 runs, it depends on all upstream instances from 10:00 to 14:00.

  • When the downstream instance at 13:00 runs, the dependency window automatically "slides" to 11:00 to 15:00.

• Applicable scenarios:

  • Cross-time-zone data aggregation: A daily task in China (UTC+8) needs to aggregate data from all 24 hourly tasks in India (UTC+5:30). You can configure the range as [-3h, 21h] to precisely cover the physical time window corresponding to India's local business day.

  • Cross-day window aggregation: A daily task that runs in the early morning needs to process data ranging from the previous afternoon to the current early morning. For example, you can configure the range as [-12h, 4h] to depend on upstream hourly instances from 12:00 yesterday to 04:00 today.

3. Specified set (depend on a group of discrete fixed instances)

When your dependency logic is related to specific, discrete output instances from upstream, "specified set" is the best choice.

• Configuration method:

  1. After selecting the dependent node, click Configure in the Actions column and set the dependency type to Specified Set.

  2. Set the left and right offsets relative to the current node's scheduled time. The maximum allowed offset range is -1440 to 1440, in minutes.

• Description: This allows a downstream task to depend on a fixed, discrete set of upstream instances. Unlike the sliding window of "specified range", this instance set is static and identical for all downstream instances.

• Core logic example:
  A downstream task is configured to depend on the upstream instances at {02:00, 06:00, 10:00}. Whether it is the downstream's 08:00 instance or its 12:00 instance, both wait until all three specific instances complete successfully before running.
  

• Applicable scenarios:

  Depending on key milestones: The downstream task's start condition is that data at several key time points throughout the day (such as morning, midday, and evening snapshots) from upstream is ready.