Assign values to cross-node output parameters

更新时间:
复制 MD 格式

Cross-node parameters allow SQL, Shell, and Python nodes to output values that downstream nodes can directly receive and use, without relying on public storage as an intermediary.

Background

In Dataphin, node dependencies form a directed acyclic graph (DAG). Previously, nodes in a DAG could only pass status information — the running status of an ancestor node would affect its descendant nodes, but the system could not interpret or forward a node's output through the DAG. To share output, an ancestor node had to write it to public storage (typically data tables), and descendant nodes had to read it back. This approach has the following drawbacks:

  • Reading from and writing to public storage is complex, and some node types cannot perform these operations. For example, a logical table node cannot read messages from public storage because its computing logic is fully determined by its field computing logic.

  • Public storage is inefficient for message passing. If multiple descendant nodes need the same message, each must read it independently. If the storage location or format changes, every descendant node must update its reading method.

  • The overhead is disproportionate to the typically small messages being passed.

To resolve these issues, Dataphin introduced cross-node parameters for passing messages between nodes.

  • Output nodes: Nodes that output parameters. Only Shell, Python, and SQL compute nodes support parameter output. Examples include MaxCompute SQL.

  • Input nodes: Nodes that receive and use parameters. An output node can connect to multiple input nodes. Any direct descendant of an output node can reference its cross-node parameters.

Instructions for assigning values to cross-node output parameters

After you declare cross-node output parameters, assign values to them in your code. For more information, see Best practices for cross-node parameters.

Important
  • Only Shell, Python, and SQL compute nodes can define cross-node output parameters.

  • Cross-node variable parameters can be used for the input parameters of integration nodes, compute nodes, and standard or hierarchical logical dimension and fact tables.

Task type

Assignment statement

Right-click menu shortcut

Shell Task

Use the following command to assign a value to a cross-node parameter:

setv "{parameter_name}" {parameter_value}

Example:

setv "output_variable" 123

Important
  • The setv "{variable_name}" {value} command assigns a value to the cross-node parameter {variable_name}. setv is a Shell function and must follow Shell syntax.

  • setv is a system-reserved function. Avoid naming conflicts.

In the editor, right-click and select Set Cross-node Parameter.

Python Task

Use the following command to assign a value to a cross-node parameter:

setv("{parameter_name}", {parameter_value})

Example:

setv("output_variable", 123)

Important
  • The setv("{variable_name}", {value}) command assigns a value to the cross-node parameter {variable_name}. setv is a Python function and must follow Python syntax.

  • setv is a system-reserved function. Avoid naming conflicts.

In the editor, right-click and select Set Cross-node Parameter.

SQL Job

set dataphin.task.result.type = dp_context_param;

select "value" as variable_name; or another query statement. The system uses the first row of the query result. The field name is the parameter name, and the field value is the parameter value.

In the editor, right-click and select Set Cross-node Parameter.