DocsICP FilingConsole
官方文档
首页

Do-while node

更新时间：
复制 MD 格式
产品详情
我的收藏

DataWorks provides the do-while node, which implements "execute first, then check" loop logic. This node is ideal when you need to repeat a series of tasks until a dynamic condition is met, such as polling to check for a file's existence or processing data in batches until the source is empty. You can orchestrate a task flow within the loop body and use a dedicated End node to control the loop's exit.

Use cases

In data development, the do-while node simplifies the design of workflows that need to repeatedly execute tasks based on a condition. Common use cases include:

  • API polling: Repeatedly call an API until it returns a specific status, such as SUCCESS, or until the response data meets your requirements.

  • Data readiness checks: Wait for an upstream file or data partition to be generated. The loop checks for the data in each iteration and exits once it is available, triggering downstream tasks.

  • Batch data processing: Process a list of items, such as table names or date partitions, provided by an upstream node like an assignment node. The do-while node iterates through the list, processing one item at a time until the list is exhausted.

  • Status synchronization: Repeatedly check the status of an external service. Once the service status changes to a desired state, such as "Available" or "Complete", the loop exits and triggers the subsequent data synchronization or processing workflow.

Requirements

  • Version requirement: This feature is available only in DataWorks Standard Edition and later versions.

  • Permission requirement: Your RAM user must be a member of the target workspace with the Development or Workspace Manager role. For more information, see Add members to a workspace.

How it works

The do-while node acts as a loop container based on an "execute first, then check" mechanism.

image Start node → loop body (execute tasks) → image End node (evaluate condition)

  1. Start and execute: The loop starts at the imageStart node, and all tasks in the loop body are executed at least once.

  2. Condition evaluation: After the loop body completes, the imageEnd node runs. You write your conditional logic in this node.

  3. Loop decision:

    • If the imageEnd node outputs the string True, a new iteration begins.

    • If the imageEnd node outputs the string False, the loop terminates, and the entire do-while node run is considered successful.

Key feature: The loop body always executes at least once, regardless of the initial condition.

Node components

Double-click a do-while node to open its internal canvas, which consists of three main parts:

  • image Start node: The entry point of the loop. It cannot be edited or deleted.

  • Loop body: A customizable canvas for your tasks. You can click Create Internal Node to add various types of internal nodes, such as Shell, SQL, and Python, to form the repeatable business process.

  • image End node: The decision-making node of the loop. It is an assignment node that uses ODPS SQL, Shell, or Python to output a True or False string. This string determines whether the loop continues.

Built-in variables

In the loop body and End node of a do-while node, you can use the following built-in variables to access the current loop state and input data:

Variable

Description

System built-in variables

${dag.loopTimes}

The current iteration number, starting from 1.

${dag.offset}

The offset of the current iteration, starting from 0. It is equivalent to ${dag.loopTimes} - 1.

Used with an assignment node
(Assuming the input parameter is bound to the output of an upstream assignment node)

${dag.input}

Returns the entire result set from the upstream node, which is typically a two-dimensional array.

${dag.input.length}

Returns the number of rows (elements) in the result set.

${dag.input[${dag.offset}]}

Retrieves the row of data that the current loop is processing and returns it as a comma-separated string, such as "user_A,beijing".

${dag.input[i][j]}

Returns the value at row i and column j (zero-based indexing).

Example: ${dag.input[0][1]} extracts the value "beijing" from the second column of the first row.

Limitations

  • Edition requirement: This feature is available only in DataWorks Standard Edition and later versions.

  • Iteration limit: The default maximum number of iterations is 128, and can be adjusted to a maximum of 1024. Exceeding this limit causes the task to fail.

  • Execution model: The node uses a serial execution model, which means each iteration must be completed before the next one begins.

  • Debugging limitation: You cannot run and test a do-while node directly in DataStudio. You must first publish the workflow and then use the Backfill Data feature in Operation Center to verify its behavior.

  • Data dependencies: If the loop depends on an upstream assignment node, be sure to backfill data starting from the upstream node to ensure the data lineage is complete.

  • Flow control: If you use a branch node within the loop body, ensure all branches eventually converge at a merge node to prevent a broken workflow.

Procedure: Create a simple loop task

This example shows how to create a task that loops five times. In each iteration, the task prints the current loop number.

Step 1: Configure the loop body

  1. Create a do-while node in your workflow.

  2. Double-click the node to open its internal canvas. In the loop body area, click Create Internal Node, select Shell, and name the node print_loop_times.

  3. Right-click the Shell node and select Open Node.

  4. In the code editor, enter the following command:

    # Use the built-in variable ${dag.loopTimes} to get the current loop number
echo "This is loop number: ${dag.loopTimes}"

  5. Click the save icon to save your changes.

Step 2: Define the exit condition

  1. Return to the do-while internal canvas. Right-click the image End node and select Open Node.

  2. Switch the language to Python.

  3. Enter the following Python code:

    # If the loop count is less than 5, continue the loop; otherwise, exit.
if ${dag.loopTimes} < 5:
    print True
else:
    print False

  4. Save the End node.

Step 3: Publish, run, and verify

  1. Return to the main workflow canvas and click the Publish button in the toolbar to publish the entire workflow.

  2. Go to Operation Center and locate the do-while node.

  3. Right-click the node and select Backfill Data > Current Node to start a test run.

  4. After the instance runs successfully, right-click it again and select View Internal Nodes.

  5. Review the five iteration records. Expand one of the iterations, for example, the fifth one. Right-click the internal print_loop_times instance and select View Runtime Log. You should see the following output:

    This is loop number: 5

Advanced use case: Process a data list

A common and practical pattern is to use a do-while node to iterate over and process a dataset produced by an upstream assignment node. In this scenario, an upstream ODPS SQL assignment node queries two rows of user information, and the do-while node processes each row.

Step 1: Configure the assignment node

  1. Create an assignment node (for example, named assign_sql_data) and set it as an upstream dependency of the do-while node.

  2. In the node, use ODPS SQL to query data:

    SELECT 'user_A', 'beijing'
UNION ALL
SELECT 'user_B', 'shanghai';

  3. Save the node. The outputs parameter will contain a two-dimensional array with two rows of data.

Step 2: Configure the do-while node

  1. In the Scheduling Configurations panel on the right side of the do-while node, find the Input Parameters section.

  2. Click Add and configure the following parameter:

    • Name: input (customizable)

    • Value Source: Select assign_sql_data.outputs

  3. Create a new Shell node in the loop body of the do-while node and enter the following script to process the current row:

    # Get the data row being processed in the current iteration
echo "Processing data row: ${dag.input[${dag.offset}]}"

  4. Open the image End node and use Python to set the exit condition:

    # Continue looping if there are more data rows to process.
if ${dag.loopTimes} < ${dag.input.length}:
    print True
else:
    print False

  5. Publish the workflow and, in Operation Center, backfill data starting from the assign_sql_data node to ensure that data flows correctly into the loop.

After the run is successful, you can check the runtime logs to see that the two iterations processed "user_A,beijing" and "user_B,shanghai" respectively.

Appendix: Do-while vs. for-each nodes

Feature

Do-while node

For-each node

Core logic

Condition-driven: Repeats execution until a condition is no longer met.

Data-driven: Executes once for each element in an input list.

Number of iterations

Indeterminate, depends on when the condition is met.

Determinate, equal to the number of elements in the input list.

Execution guarantee

Executes at least once, even if the initial condition is not met.

If the input list is empty, it does not execute at all.

Use cases

Polling, waiting, status checks, and batch processing until a source is empty.

Batch processing a known list, such as synchronizing a set of tables or processing multiple partitions.

Control method

The End node outputs True or False to control whether the loop continues.

Automatically ends after iterating through all elements.

Previous：for-each nodeNext：Check node