Using a branch node for time-specific tasks

Background

You cannot use a cron expression to schedule a node to run only on the last day of each month. The branch node lets you implement this logic with a switch-case model. For more information, see branch node.

Branch nodes and other control nodes

On the DataStudio page, you can find the control nodes supported by your DataWorks edition, such as the assignment node, branch node, and merge node.

• ASN: An assignment node that performs complex calculations. Its output determines which branch condition is met.

• X and Y: Branch nodes that are downstream of the ASN assignment node. They select branches based on the output of ASN. As shown by the green lines in the figure, node X selects the left branch, and node Y selects the two leftmost branches:

  • Nodes A and C run because they are downstream of selected outputs from both X and Y.

  • Node B is set to dry run. Although it is downstream of a selected branch of node Y, it is not on a selected branch of node X.

  • Node E is set to dry run because it is not on a selected branch of node Y, even though it has an upstream standard node Z.

  • Node G is set to dry run because its upstream node E is in a dry run state, even though nodes C and F run.

  • When does the dry run state stop propagating downstream?

    The JOIN node is a merge node. It stops the dry run state from propagating. Because node D is downstream of the JOIN node, the dry run state from node B is blocked, and node D runs.

Using a branch node together with other control nodes, you can create a workflow where a specific node runs only on the last day of each month.

Use a branch node

1. Define node dependencies.

   1. The root assignment node uses the scheduling time SKYNET_CYCTIME to determine if the current day is the last day of the month. If it is, the node outputs 1. Otherwise, the node outputs 0. This output is captured by DataWorks and passed to the downstream.

   2. The branch node uses the output of the assignment node to define its branches.

   3. Two Shell nodes, downstream of the branch node, execute different branch logic.

2. Define the assignment node.

   When you create an assignment node, it includes a built-in outputs parameter. You can write code for an assignment node in SQL, Shell, or Python.

   • For SQL nodes, DataWorks captures the result of the last SELECT statement as the value of the outputs parameter.

   • For Shell and Python nodes, DataWorks captures the last line of the standard output as the value of the outputs parameter.

   This example uses Python for the assignment node. The following figures show the code and scheduling property settings.

   • Code

   • Configuration of scheduling properties

3. Define the branches.

   You can use Python expressions to define conditions for a branch node. Each condition is bound to an output. When a condition is met, the downstream nodes that depend on the corresponding output are executed. All other downstream nodes are set to dry run.

   • Scheduling configuration

   • Branch configuration

   • Outputs bound to conditions generated in the scheduling configuration

4. Attach task nodes to different branches.

   The branch node in this example has three outputs. You can select any output as an input for a downstream node. Because each output corresponds to a condition, choose carefully.

   • Dependency for the node that runs on the last day of the month

   • Dependency for the node that runs on other days of the month

5. Verify the result.

   After you configure the nodes, commit and deploy them. Then, you can backfill data to test the workflow. Set the data timestamps to 2018-12-30 and 2018-12-31, which correspond to scheduling times of 2018-12-31 and 2019-01-01, respectively. The first backfill instance triggers the last-day logic, and the second instance triggers the non-last-day logic. The following sections show the results.

   Data timestamp: 2018-12-30 (scheduling time: 2018-12-31)

   • Branch selection result for the branch node

   • The node for the last day runs.

   • The node for other days is set to dry run.

   Data timestamp: 2018-12-31 (scheduling time: 2019-01-01)

   • Branch selection result for the branch node

   • The node for the last day is set to dry run.

   • The node for other days runs.

Summary