DocsICP FilingConsole
Documentation
Home Page DataWorks User Guide Data Studio (new version) Node development Node types EMR EMR Shell node

EMR Shell node

Updated at:
Copy as MD
Product
My Favorites

EMR Shell nodes in DataWorks run custom Shell scripts for advanced operations such as data processing, calling Hadoop components, and file management.

Prerequisites

  • Before you start node development, if you need a custom component environment, create a custom image based on the official dataworks_emr_base_task_pod image and then use it in Data Studio. For more information, see Create a custom image and Use images in data development.

    For example, you can replace Spark JAR packages or add dependencies on specific libraries, files, or JAR packages when you create a custom image.

  • Create an Alibaba Cloud EMR cluster and register it with DataWorks. For more information, see New Data Studio: Attach an EMR compute resource.

  • (Optional, for RAM users) The Resource Access Management (RAM) user for task development must be added to the workspace and assigned the Development or Workspace Administrator role (this role includes extensive permissions and must be granted with caution). For more information, see Add workspace members.

    If you are using a root account, skip this step.

Limitations

  • EMR Shell nodes can run only on a serverless resource group (recommended) or an exclusive resource group for scheduling. Using a custom image for data development requires a serverless resource group.

  • To manage metadata for a DataLake cluster or a custom cluster in DataWorks, you must first configure EMR-HOOK on the cluster. For more information, see Configure EMR-HOOK for Hive.

    Note

    If EMR-HOOK is not configured on the cluster, you cannot view metadata in real time, generate audit logs, display data lineage, or perform EMR-related governance tasks in DataWorks.

  • For tasks submitted with spark-submit, set the deploy-mode to cluster. The client mode is not recommended.

  • EMR Shell nodes run on DataWorks scheduling resource groups, not on EMR clusters. You can use some EMR component commands, but you cannot directly read the resource status from the EMR cluster. Therefore, you must first upload any required resources to DataWorks. For more information, see Resource Management.

  • EMR Shell nodes do not support running Python files. Use a Shell node to run them.

Procedure

  1. In the EMR Shell node editor, perform the following operations.

    Develop Shell code

    Choose an option based on your requirements:

    Option 1: Upload and reference EMR JAR

    Upload resources from your local machine to Data Studio and reference them. For a DataLake cluster, follow these steps to reference an EMR JAR resource. If a resource is too large to upload through the DataWorks UI, store it in HDFS and reference it in your code.

    1. Create an EMR JAR resource.

      1. For more information, see Resource Management. Store the JAR package generated in Prepare initial data and a JAR package in the JAR resource directory emr/jars. Click Click Upload.

      2. Select a Storage Path, Data Sources, and Resource Group.

      3. Click Save.

      For Storage Path, select HDFS.

    2. Reference the EMR JAR resource.

      1. Open the created EMR Shell node and go to the code editor page.

      2. In the left-side navigation pane, find the resource you want to reference under Resource Management (for example, onaliyun_mr_wordcount-1.0-SNAPSHOT.jar). Right-click the resource and select Insert Resource Path.

      3. After you select a reference, a success message appears on the code editing page of the EMR Shell node, indicating the resource is referenced successfully. Run the following command, replacing the resource package, bucket name, and path with your actual values.

      ##@resource_reference{"onaliyun_mr_wordcount-1.0-SNAPSHOT.jar"}
onaliyun_mr_wordcount-1.0-SNAPSHOT.jar cn.apache.hadoop.onaliyun.examples.EmrWordCount oss://onaliyun-bucket-2/emr/datas/wordcount02/inputs oss://onaliyun-bucket-2/emr/datas/wordcount02/outputs
      Note

      Comments are not supported when you edit code in an EMR Shell node.

    Option 2: Directly reference OSS resource

    Reference OSS resources directly by using OSS REF. When the EMR node runs, DataWorks automatically loads the OSS resources to the local machine.

    ossref://{endpoint}/{bucket}/{object}

    • Endpoint: The public access endpoint for OSS. If you leave this parameter empty, the OSS bucket must be in the same region as the EMR cluster.

    • bucket: The name of the bucket, which is a container that stores objects in OSS. Each bucket has a unique name. You can log on to the OSS console to view all buckets under the current account.

    • object: The specific object, such as a file name or path, stored in the bucket.

    Example

    1. Upload a sample file to an OSS bucket. In this example, we use the emr_shell_test.sh file with the following content:

      #!/bin/sh
echo "Hello, DataWorks!"

    2. Directly reference the OSS resource in the EMR Shell node.

      sh ossref://oss-cn-shanghai.aliyuncs.com/test-oss-of-dataworks/emr_shell_test.sh
      Note

      In this command, oss-cn-shanghai.aliyuncs.com is the endpoint, test-oss-of-dataworks is the bucket name, and emr_shell_test.sh is the object file name.

      The command returns the following output from the emr_shell_test.sh file:

      ...
>>> [2024-10-24 15:46:01][INFO   ][CommandExecutor       ]: Process ready to execute. command: sh ./emr_shell_test.sh
>>> [2024-10-24 15:46:01][INFO   ][CommandExecutor       ]: Command state update to RUNNING
>>> [2024-10-24 15:46:01][INFO   ][CommandExecutor       ]: Process start to execute...
Process Output>>> Hello, DataWorks!
...

    Configure EMR Shell scheduling parameters

    Define variables in the Shell editor using the ${variable_name} format, then assign values in the Scheduling Parameters section under Scheduling Settings on the right side of the node editor. This passes parameters dynamically during scheduled runs. For more information, see Sources and expressions of scheduling parameters. Example:

    DD=`date`;
echo "hello world, $DD"
## Can be used with scheduling parameters
echo ${var};
    Note

    If you are using a DataLake cluster, the following command-line tools are also supported.

    • Shell commands: Shell commands in /usr/bin and /bin, such as ls and echo.

    • YARN components: hadoop, hdfs, and yarn.

    • Spark components: spark-submit.

    • Sqoop components: sqoop-export, sqoop-import, sqoop-import-all-tables, and more.

    If you use these components to access an RDS instance, you must add the resource group's IP address to the RDS allowlist.

    Run the Shell task

    1. In the Run Configuration section, configure the Compute Resource and Resource Group.

      Note

      • You can also CUs for Scheduling based on the resources required for task execution. The default CU is 0.25.

      • You can select an Image based on your task requirements.

      • To access a data source in a public network or a VPC environment, you must use a scheduling resource group that can connect to the data source. For more information, see Network connectivity solutions.

    2. In the toolbar, click Run to run the Shell task.

  2. To run the node periodically, configure its scheduling properties. For more information, see Configure scheduling properties for a node.

    Note

    If you need to customize the component environment, you can create a custom image based on the official image dataworks_emr_base_task_pod and use the image in Data Development.

    For example, when you create a custom image, you can replace Spark JAR packages or depend on specific libraries, files, or JAR packages.

  3. Deploy the node. For more information, see Deploy nodes and workflows.

  4. View the running status in Operation Center. For more information, see Get started with Operation Center.

FAQ

Q: A hosts mapping has been configured on the legacy scheduling resource group, but the EMR Shell node still reports a resolution failure. How do I resolve this?

A: Reinitialize the EMR cluster using the configured resource group so that the EMR Shell script recognizes the new hosts mapping. Go to the compute resources list page, click Initialize Resources, and then click Re-initialize in the dialog box.

Related documentation

Previous：EMR PySpark nodeNext：EMR Spark node