DocsICP FilingConsole
Documentation
Home Page DataWorks User Guide Data Studio (new version) Node development Node types General nodes Shell node

Shell node

Updated at:
Copy as MD
Product
My Favorites

The DataWorks shell node is designed for data engineers. It runs standard shell scripts for file operations and data transfer with OSS or NAS. You can configure scheduling parameters, reference resources, use the preinstalled ossutil tool to access OSS securely, and run node tasks under an associated RAM role.

Permissions

Add the RAM account used for node development to the target workspace and grant it the developer or workspace administrator role. For details, see Add a member to a workspace.

Usage notes

  1. Syntax limitations

    • Standard shell syntax is supported. Interactive syntax is not supported.

  2. Runtime environment and network access

    • When you run a shell node on a serverless resource group, if the target service has an allowlist configured, you must add the IP addresses of the serverless resource group to the allowlist of the target application.

    • When you use a serverless resource group, a single task supports a maximum of 64CU. To prevent resource shortages and task startup delays, do not exceed 16CU.

  3. Extended development environment

    • If your task requires a specific development environment, you can use the custom image feature in DataWorks to build a tailored image for task execution. For more information, see Use custom images.

  4. Resource usage and multi-script calls

    • Avoid launching a large number of child processes within a shell node. Because shell nodes currently have no resource limits, excessive child processes can affect other tasks running on the same scheduling resource group.

    • If a shell node invokes another script, such as a Python script, the shell node waits for the invoked script to complete before the node itself exits.

Quick start

This section walks you through the process of creating, debugging, configuring, and deploying a shell node by using an example that outputs "Hello DataWorks!".

Develop the node

  1. Log on to the DataWorks console. After you switch to the target region, in the navigation pane on the left, click Data Development and O&M > Data Development. From the drop-down list that appears, select the target workspace and click Data Analytics.

  2. On the Data Studio page, create a shell node.

  3. Enter standard shell code in the script editor (interactive syntax is not supported):

    echo "Hello DataWorks!"

  4. After you finish developing the code, click Debug Configuration on the right side, select the target debugging resource group and other required execution parameters, and then click the imageRun button to trigger a local debugging run.

  5. After the script passes debugging verification, click Scheduling Settings on the right side to configure production-level scheduling policies, including the schedule, dependencies, and parameter settings, so that the node can be automatically executed on a periodic basis as planned.

  6. After you complete the task scheduling configuration, you must click Save before you can proceed to the next step.

Deploy and operate the node

  1. After you complete the schedule settings, you can submit and deploy the shell node to the production environment to transition it from the development state to the production state. For the specific procedure and deployment process, see Deploy a node.

  2. After the node is deployed, the task runs periodically based on the schedule you configured. You can click the image icon in the upper-left corner, and in the navigation page that appears, choose All Products > Data Development and O&M > Operation and Maintenance Center to go to Operation Center. Then, in the left-side navigation pane, go to Node O&M > Auto Triggered Task O&M > Auto Triggered Task to view the deployed scheduled tasks. For more information, see Manage scheduled tasks.

Advanced usage

Reference resources

  1. DataWorks allows you to upload resources that you need in shell nodes to DataWorks through resource management. For details, see Manage resources.

    Note

    Resources must be deployed before they can be referenced by nodes. If a production task requires a resource, you must also deploy the resource to the production environment.

  2. Open the shell node that you created to go to the script editor page.

  3. Click image in the left-side navigation pane to open the resource management menu. Find the resource that you want to use, right-click it, and select Reference Resource. You can then reference the resource in the shell script.

    Note

    • After the resource is referenced, the system automatically inserts a declaration comment in the format of ##@resource_reference{resource_name} at the top of the script.

    • This comment is required for DataWorks to identify resource dependencies and automatically mount the corresponding resource to the execution environment at runtime. Do not manually modify or delete it.

Use scheduling parameters

Scheduling parameters are injected into shell nodes as positional parameters. Custom variable names are not supported. DataWorks passes the parameter values that you configure in Scheduling Settings > Scheduling Parameters to the shell script as positional parameters such as $1, $2, and $3, in order. When the number of parameters exceeds 9, use the brace syntax such as ${10} and ${11} to ensure correct parsing. Separate multiple parameter values with spaces, and make sure that the order strictly matches the positional references in the script.

For example, in this example:

  • Parameter $1 is assigned the current time: $[yyyymmdd].

  • Parameter $2 is assigned a fixed value: Hello DataWorks.

Note

  • If a parameter value contains spaces, enclose it in quotation marks. The entire content within the quotation marks is treated as a single parameter.

  • To obtain the output parameters of an upstream node, add parameters in Scheduling Settings > Node Context Parameters > Input Parameters and set the parameter values to the output parameters of the upstream node.

Use ossutil to access OSS

The DataWorks shell node natively supports the Alibaba Cloud OSS command-line tool ossutil, which allows you to manage buckets, upload and download files, and perform batch operations. You can configure access credentials by using a configuration file or command-line parameters to access OSS with ossutil.

In the new Data Studio, for shell nodes specifically, in addition to the preceding two methods, you can associate a RAM role and use Alibaba Cloud STS (Security Token Service) to dynamically obtain temporary security credentials for the role, and then use ossutil to access OSS. This method allows you to securely access target cloud resources without hardcoding long-term AccessKey credentials in your code. For details, see Associate a RAM role to run a node.

Note

ossutil is pre-installed in the DataWorks environment. You do not need to manually install it. The default path is /home/admin/usertools/tools/ossutil64.

Use datasets to access OSS or NAS

DataWorks allows you to create OSS or NAS datasets and then reference the datasets in shell node development to read from and write to OSS or NAS storage during shell node execution.

Run a node by using an associated role

You can associate a RAM role to run a node, which allows you to run node tasks with a specific RAM role for fine-grained permission control and security management.

Appendix: Script exit codes

You can use script exit codes to further determine whether a script ran successfully.

  • Exit code 0: Indicates success.

  • Exit code -1: Indicates the process was terminated.

  • Exit code 2: Indicates that the platform needs to automatically rerun the task once.

  • Other exit codes: Indicate failure.

The following is a sample runtime log when the shell node runs successfully (exit code 0).

INFO  Exit code of the Shell command 0
INFO  --- Invocation of Shell command completed ---
INFO  Shell run successfully!
Note

Due to the underlying shell mechanism, the exit code of the entire script in a shell node equals the exit code of the last executed command.

Previous：Assignment nodeNext：Parameter node