A SUB_PROCESS node is a special task type for referencing another workflow. This is ideal for breaking down complex processes into smaller, manageable sub-workflows, which can be defined and managed independently to improve maintainability and reusability.
Background
You can nest workflows by using a SUB_PROCESS node. A referenced workflow does not require its own scheduled task or dependency relationship settings. The referencing workflow controls its execution.
-
When it runs: A workflow set as Referenceable runs only when another workflow references it.
-
Execution time: A workflow set as Referenceable runs based on the execution time of the referencing workflow.
In a multi-level nested structure, the scheduling system starts execution from the deepest level of reference (for example, the fourth level in the left diagram). During execution, if the system detects a SUB_PROCESS node in the current workflow, it synchronously triggers the referenced workflow. For example, as shown in the left diagram, Workflow-E contains a SUB_PROCESS node that references Workflow-D. Therefore, when Workflow-E runs, it synchronously triggers Workflow-D.
-
Number of executions: A workflow set as Referenceable runs once for each time it is referenced.
If a workflow contains multiple SUB_PROCESS nodes that reference the same target workflow, that target workflow runs once for each reference.
Depending on your business needs, you can build a simple linear reference structure as shown in the left diagram, or design a more complex, multi-branch hierarchy that allows for parallel execution, as shown in the right diagram. Each structure has its own use case. Choose the one that best fits your business process:
-
Left diagram: This diagram shows a linear workflow hierarchy where each workflow depends on the previous one and references it through an internal SUB_PROCESS node. For example, Workflow-E references Workflow-D, which in turn references another workflow, and so on, until Workflow-B references Workflow-A.
-
Right diagram: This diagram shows a complex workflow hierarchy where a single workflow can be referenced by multiple workflows through their internal SUB_PROCESS nodes. For example, Workflow-A is referenced by both Workflow-B1 and Workflow-B2.
Prerequisites
You have a workspace with Use Data Studio (New Version) enabled and a resource group attached. For more information, see Create a workspace.
Limits
-
When you use SUB_PROCESS nodes for multi-level nesting, the maximum nesting depth is
5levels (including the root workflow), and the total number of workflows cannot exceed200. -
The workflow reference created by a SUB_PROCESS node is a trigger relationship, not an upstream or downstream dependency relationship. When a SUB_PROCESS node is detected in a workflow, it triggers the execution of the referenced workflow.
-
For a workflow triggered by a reference, the values of its scheduling parameters depend on the scheduled time of the triggering workflow.
-
After you set a workflow as Referenceable, neither the workflow nor its internal nodes can depend on tasks outside the workflow, nor can they be depended on by external tasks, including the workspace root node.
Configure workflows
This topic describes how to use a SUB_PROCESS node to reference an existing workflow. The following steps provide an example in which workflow Workflow2 references workflow Workflow1.
Workflow 1: Enable Referenceable
To make a workflow referenceable, you must remove its upstream and downstream dependencies. A workflow set as Referenceable cannot be scheduled to run on its own. Use the following steps to enable the Referenceable switch for a workflow. We recommend that you create a new workflow to try this feature.
You can follow the steps below to create a Referenceable workflow Workflow1 and enable the Referenceable switch for workflow Workflow1.
-
Create a workflow
Workflow1.Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the desired workspace and choose in the Actions column.
-
In the left navigation pane, click
. To the right of Project Directory, click . Set a name for the workflow, save it, and then the workflow editor opens.
-
Turn on the Referenceable switch.
Go to the edit page for the
Workflow1workflow, click General on the right side of the page, and in the Properties configuration dialog box, turn on the Can be cited switch.Note-
After you enable the Referenceable capability, the current workflow can be referenced by other workflows by using a SUB_PROCESS node. You do not need to configure scheduling time, scheduling dependencies, or other similar properties.
-
After the Referenceable capability is enabled, neither the workflow itself nor its internal nodes can depend on tasks outside the workflow, nor can they be depended on by external tasks. This includes the workspace root node.
-
Workflow 2: Reference Workflow 1
You can create a SUB_PROCESS node in the workflow to reference the Workflow2Workflow1 workflow.
-
Create the workflow
Workflow2. For more information, see Scheduled Workflow Orchestration. -
Create a
SUB_PROCESSnode.Click the workflow
Workflow2that you created to open the workflow editing page. Drag the SUB_PROCESS node from under Logical Node to the canvas. -
Configure the referenced workflow.
In the Create Node dialog box, set Reference Workflow to Select existing. In the drop-down list below Select existing, select the workflow that you want to reference,
Workflow1. For Node Name, you can use the system-generated name or enter a custom name. Click Confirm to complete the reference.Note-
If you need to reference a Create workflow, you can enter a custom Workflow Name and click Confirm to complete the reference.
-
After confirming the reference to a new workflow, you can hover over the SUB_PROCESS node and click Open Reference Workflow. In the prompt that appears, click Save and Open to open the new workflow's editor.
-
On the editor page of the newly created referenced workflow, the Can be cited switch in the General panel on the right is enabled by default.
-
-
Save and view reference details.
Go to the edit page of workflow
Workflow2, click Save at the top, and on the CHANGE CHECK page, complete the Save confirmation.NoteAfter the Referenceable switch is turned on for a workflow, it can be referenced by multiple workflows. You can view the number of references in the General panel on the right side of the referenced workflow's page. Click View Details for more information.
Next steps
After you configure the workflow reference, you can publish the workflow to run periodically and use Data Backfill to monitor its status.
-
Configure scheduling.
If a workflow task needs to run periodically, first configure its scheduling properties. For more information, see Configure scheduling properties for a node.
-
Publish the workflow task.
A workflow must be published to the production environment to be automatically scheduled. For more information, see Scheduled Workflow Orchestration.
ImportantWhen you publish
Workflow2, which contains theSUB_PROCESSnode, you must first publishWorkflow1, the workflow referenced by theSUB_PROCESSnode. Otherwise, the publication will fail. -
Run and check the workflow task.
After publishing the workflow task, use the Data Backfill feature in Operation Center to check the status of the scheduled run. For more information, see Get started with Operation Center.
Find the running SUB_PROCESS node, right-click it, and select View Tasks That Reference Workflow to directly view the execution status of all internal node tasks within the referenced workflow.
> Create Workflow