Configure same-cycle scheduling dependencies

Node output

A node output, also known as the output name of the current node, identifies the node for other nodes to depend on. It does not represent the actual data the node produces. Other nodes use this output name to specify it as an ancestor dependency.

DataWorks automatically generates two output names for each node:

• projectName.randomNumber_out: This output is globally unique and cannot be modified or deleted.

• projectName.nodeName: This output includes the node name and can be modified. This output name remains unchanged even if the node is renamed.

DataWorks also allows you to add outputs manually or by automatically parsing inputs and outputs from your code. Support for automatic parsing varies by node type. For details, see Comparison of automatic parsing results.

Node input

A node input specifies the ancestor nodes that the current node depends on. You can specify a dependency by using the ancestor's output name (recommended), node name, or node ID.

A node ID is generated only after the ancestor node is submitted to the production environment.

Configure node dependencies by parsing table lineage from code

Automatic parsing analyzes the table lineage in a node's code and automatically configures the node's output names and upstream dependencies. After parsing, tables that the node writes to are automatically added as node outputs in the projectname.tablename format, and tables that the node reads from are automatically added as node inputs.

For example, when a node uses SELECT on a table, that table is automatically parsed as an upstream dependency of the node. When a node uses INSERT on a table, that table is automatically parsed as an output of the node. For the keywords supported by automatic parsing for each node type, see Keywords supported by automatic parsing for each node type.

• Configure dependencies

  Automatic parsing supports two methods: Parse Inputs and Outputs from Code and Automatic Parsing Before Committing. Both methods work on the same principle. Automatic parsing before committing automatically parses inputs and outputs when you submit the code and prompts you to configure dependencies.

  For example, ODPS node mc2 depends on the output table dws_user_info_all_di of node mc1. The code of node mc2 is as follows:

  INSERT OVERWRITE TABLE ads_user_info_1d PARTITION (dt='${workflow.var}')
  SELECT uid
    , MAX(region)
    , MAX(device)
    , COUNT(0) AS pv
    , MAX(gender)
    , MAX(age_range)
    , MAX(zodiac)
  FROM dws_user_info_all_di
  WHERE dt = '${workflow.var}'
  GROUP BY uid;

  After you click Parse Inputs and Outputs from Code, the input of this node is parsed as the dws_user_info_all_di table, and the output table name and name of the upstream node are automatically matched:

  Upstream node output name	Upstream node output table name	Upstream node name	Node ID	Workspace	Owner	Schedule	Method	Recent run status	Action
  yunwan_lingyi.dws_user_info_all_di	yunwan_lingyi.dws_user_info_all_di	mc1	-	Test workspace	lingyi01_testcloud_com	Day	Code parsing	No data	Delete

  At the same time, the output of this node is parsed as the ads_user_info_1d table. The parsing results are as follows:

  Output name	Output table name	Downstream node name	Owner	Method	Downstream node affected baselines	Action
  old_ide.505487297_out	-	-	-	Added by system	-	Delete
  old_ide.mc2	-	-	-	Manually added	-	Delete
  yunwan_lingyi.ads_user_info_1d	yunwan_lingyi.ads_user_info_1d	-	-	Code parsing	-	Delete

  Node mc2 is now configured with a dependency on node mc1.

• Modify dependencies from code parsing

  When the dependencies from code parsing do not meet your expectations, or when scenarios that do not support scheduling dependencies (tables whose data is not produced by periodic scheduling) require you to manually remove dependencies, refer to the following instructions to modify the automatically parsed dependencies.

  Action	Description
  Manually delete parsing results	Delete the unexpected input from the upstream node dependency list, perform the delete operation, and re-parse. After deletion, a corresponding comment is automatically added to the code to prevent the dependency from being re-added during the next parsing: --@exclude_input=Remove input --@exclude_output=Remove output
  Manually add inputs and outputs	Right-click a table name in the code editor and select Add Input or Add Output. After the input or output is added, a corresponding comment is automatically added to the code. --@extra_output=Add output --@extra_input=Add input Alternatively, you can add dependencies by using the methods described in Manually add upstream node dependencies from the schedule settings panel or Set node dependencies by dragging connections in the workflow panel.

  Important

  DataWorks does not allow you to directly delete a node output that has existing downstream dependencies. Doing so will cause downstream task execution or data retrieval errors. We recommend that you first adjust the downstream business logic, remove the upstream dependency from the downstream node, and then delete the node output from the upstream node.

• Scenarios excluded from automatic parsing

  Temporary tables defined in workspace table management with a fixed naming format (for example, tables prefixed with t_) in DataWorks are not automatically parsed as node outputs or upstream dependencies.

• Considerations for automatic parsing

  When you use automatic parsing to configure dependencies, make sure that node outputs are unique within the current region. When developing in DataWorks, note the following:

  • Node creation: Each node has a default output with the same name as the node. If nodes with the same name exist in the same workspace, you must manually modify the node output of one of them.

  • Code development: Automatic parsing uses the output table of a node as the node output. If two scheduled nodes in the same workspace insert data into the same table, automatic parsing will cause an error for one of the nodes. For more information, see Multiple nodes write data to the same table, and automatic parsing reports duplicate node output names.

  • Dependency configuration: When you use SQL tasks to process the output tables of batch synchronization tasks, to enable SQL tasks to quickly depend on batch synchronization tasks through lineage-based automatic parsing, you must manually configure the output table of the batch synchronization node as a node output, or name the batch synchronization task node after its output table (the platform automatically creates a node output with the same name as the node). Otherwise, when submitting the downstream SQL node, you may encounter the error The parent node output name ${projectname.tablename} that the current node depends on does not exist. The current node cannot be submitted. Make sure that the parent node with this output name has been submitted.

Manually add upstream node dependencies from the schedule settings panel

In the Schedule Settings > Scheduling Dependency > Parent Nodes section, manually add upstream node dependencies by entering the node output, node name, or node ID of the target node. Because node names can be duplicated, we recommend using node outputs to configure dependencies.

Set node dependencies by dragging connections in the workflow panel

When you set dependencies by dragging connections in the DAG panel of a workflow, DataWorks automatically adds the upstream node's _out output to the downstream node, creating a node dependency.