Cross-node parameters-Dataphin(Dataphin)-阿里云帮助中心

Scenario

A retail company needs to check a supplier's service each day for product catalog updates. If the catalog has changed, the company pulls the entire new catalog. This is a costly operation because of the large data volume. If there are no changes, the company uses its existing data. How can you implement this process in Dataphin?

Solution and features

You can write a program that checks for product catalog changes and place it in an output node. This node outputs the update status as a cross-node parameter. A separate input node syncs the product catalog. This input node uses conditional scheduling based on the value of the cross-node parameter.

Output node

Go to R&D > Data Studio > Compute Task. Create a Python node to check for product catalog changes. Add the cross-node output parameter update_status.
Enter the Python code for the node. Use a random function to return the status. This is for demonstration purposes only. In a real-world scenario, you must return the status based on whether the product catalog has changed. After you write the code, submit the node.
```
from random import randint

def check_update():
    return randint(0,1)
    
stev("update_status",check_update())    # Cross-node parameter
```

Input node

Create an offline pipeline node named imp_product_catalog. Add the check_update node as its upstream node.
Enable conditional scheduling. Set the scheduling condition to perform a dry-run when check_update.update_status = 0 (no update). After you complete the configuration, you can preview the scheduling plan. If the preview value is 0, the node performs a dry-run. If the preview value is 1, the node performs a normal run.
This process uses the cross-node parameter from the ancestor node and conditional scheduling to dynamically adjust the schedule of the descendant node.