EMR PySpark node

Updated at:

DataWorks provides EMR PySpark nodes, where you can collaboratively write Python business logic and spark-submit submission commands in a unified development interface. The nodes connect to Alibaba Cloud EMR semi-managed clusters and fully managed EMR Serverless Spark.

Scope

  • Compute resource limits:EMR PySpark nodes support these two compute resource types:

    • EMR compute resources (semi-managed clusters): only DataLake clusters and Custom clusters.

    • EMR Serverless Spark compute resources (fully managed clusters).

      Important

      To run an EMR PySpark node on an EMR semi-managed cluster, such as a DataLake or Custom cluster, submit a ticket. Include your cluster type, EMR version, and intended use case so that we can assess integration feasibility and provide support.

  • Resource group limits:Only Serverless resource groups are supported.

  • Permission limits:The Alibaba Cloud account owner or a RAM user who has been added to the workspace and has the Dev or Workspace Administrator role. For instructions on adding members, see Add members to a workspace are supported.

Create a node

  1. Log on to the DataWorks console. Switch to the target region, then click Data Development and O&M > Data Development in the left navigation pane. Select the workspace from the drop-down list and click Data Analytics are supported.

  2. On the Data Development (Data Studio) page, create an EMR PySpark node.

  3. Configure the nodePath and Name. In this example, the node is named emr_pyspark_test.

Develop the node

An EMR PySpark node uses a two-pane collaborative editor:

  • Upper pane for Python code. Write the core business logic here. You can reference uploaded resource files, such as .py modules.

  • Lower pane for Submit commands. Use it to submit spark-submit commands to the EMR cluster.

The following Monte Carlo example shows how an EMR PySpark node estimates pi in parallel.

Step 1: Upload dependency resources

Upload your custom Python file (in this example, utils.py) to the DataWorks resource management module. After the upload, you can reference this resource in your node. This Python file defines the core computation logic for Monte Carlo sampling within a single Spark task.

  1. For more information about how to upload resources, see Create and manage resources. On the resource management page in Data Studio, click Create Resource, select the resource type as EMR File, and configure the resource Name.

  2. Click Re-Upload to upload the utils.py file used in this example.

  3. Select the Storage path, Connection, and Resource Groups, and then click Save.

Step 2: Write Python code

In the Python code editor, enter the following code. The example estimates pi by dividing the total work into tasks and distributing them across cluster compute nodes. After parallel execution, it collects and aggregates the results to produce the final estimate.

##@resource_reference{"utils.py"}
from pyspark.sql import SparkSession
from utils import estimate_pi_in_task
import sys

def main():
    # Create a SparkSession
    spark = SparkSession.builder \
        .appName("EstimatePi") \
        .getOrCreate()

    sc = spark.sparkContext

    # Total number of samples
    total_samples = int(sys.argv[1])
    num_partitions = ${test1}

    # Samples per partition
    samples_per_partition = total_samples // num_partitions

    # Create an RDD and run estimate_pi_in_task once per partition
    rdd = sc.parallelize(range(num_partitions), num_partitions)

    # Map each partition to a sampling task
    inside_counts = rdd.map(lambda _: estimate_pi_in_task(samples_per_partition))

    # Aggregate results from all partitions
    total_inside = inside_counts.sum()
    pi_estimate = 4.0 * total_inside / total_samples

    print(f"Total samples: {total_samples}")
    print(f"Samples inside the circle: {total_inside}")
    print(f"Estimated pi: {pi_estimate:.6f}")

    spark.stop()

if __name__ == "__main__":
    main()
Note

You can submit only the entire Python file as a Spark job; you cannot run a selected portion of the code.

Additional information about the script parameters:

Parameter

Type

Value

Description

sys.argv[1]

Command-line argument

spark-submit The value after the script name in the

Specifies the total number of samples, for example 10000.

${test1}

Scheduling parameter

Dynamically replaced with its actual value by DataWorks during execution or scheduling

Specifies the number of partitions, for example 100.

Step 3: Write the spark-submit command

In the Submit command editor, enter the following code. The command uses the spark-submit tool to package the specified Python application and submit it to an Alibaba Cloud EMR cluster.

The parameters supported by spark-submit differ significantly between EMR compute resources (semi-managed clusters) and EMR Serverless Spark compute resources (fully managed clusters). The following table compares sample code, command integration, and reference documents to help you configure them correctly.

Comparison item

EMR compute resources (semi-managed clusters)

EMR Serverless Spark compute resources (fully managed clusters)

Sample code

spark-submit \                          
 --master yarn \            # Set YARN as the Spark cluster manager
 --deploy-mode cluster \    # Use cluster deployment mode
 --py-files utils.py \      # Explicitly declare the required Python resource file
 emr_pyspark_test.py 10000  # Main program entry file
spark-submit \                                       
 --py-files utils.py \                    
 emr_pyspark_test.py 10000  

Command integration

Based on the spark-submit tool from the Apache Spark open-source distribution

Based on the Alibaba Cloud EMR Serverless spark-submit toolkit

Command reference

For more information about parameters supported by spark-submit and how to use them, see Apache Spark documentation: Submitting Applications

For parameter syntax, common options, and best practices for spark-submit, see Submit a job by using spark-submit are supported.

Note

EMR PySpark nodes do not provide a dedicated Spark parameter configuration section. Custom Spark Conf parameters must be specified in the spark-submit command in the Submit Command pane using the --conf key=value format.

The following template parameter and engine version priorities apply only to EMR Serverless Spark compute resources, not to EMR semi-managed DataLake or Custom clusters; parameters and versions for those clusters are determined by the cluster.

When a node has multiple configuration sources, such as global Spark parameters and task-level parameters configured in the spark-submit command using --conf, Spark properties and engine versions take effect in the following priority orders (from highest to lowest):

Spark properties:

  1. Parameters in the specified Spark configuration template referenced by spark.emr.serverless.templateId.

  2. DataWorks global Spark parameters (effective when Global configuration takes priority is selected in Configure global Spark parameters).

  3. Parameters specified in the current node's spark-submit command in the Submit Command pane in the --conf key=value format.

  4. Parameters in the default Spark configuration template of EMR Serverless Spark (included in the priority order only when this default template is actually applied).

Engine versions:

  1. The engine version specified by --release-version "esr-4.1.1 (Spark 3.5.2, Scala 2.12)", for example, in the current node's spark-submit command in the Submit Command pane.

  2. The engine version specified in the Spark configuration template referenced by spark.emr.serverless.templateId.

  3. The engine version specified in the default Spark configuration template of the EMR Serverless Spark workspace (included in the priority order only when this default template is actually applied).

  4. The engine version specified by the task-level DataWorks parameter SERVERLESS_RELEASE_VERSION, if configured.

  5. The default engine version of the EMR Serverless Spark compute resource bound to DataWorks.

Run the node

  1. In Running Configurations, configure Compute Resource and Resource Group are supported.

    Setting

    Description

    Compute resource

    Select a bound EMR compute resource or or EMR Serverless Spark compute resource. If no compute resource is available, select Create Compute Resource are supported.

    Resource group

    Select a resource group bound to the workspace.

    Script parameters

    When configuring the node, define a variable in the form ${parameter_name}. Configure its Script Parameters information in Value Used in This Run. The task dynamically substitutes the actual value during execution.

    The parameter value is synchronized from the scheduling configuration.

    Note

    The value for this run applies only to this execution. The system first uses Value Used in This Run; if unavailable, it uses Parameter Value are supported.

  2. On the toolbar at the top of the node editor, click Run. The system combines the complete Python script, including resource references, and submits it to the EMR cluster through spark-submit. It then returns execution logs and results.

Next steps

  • Configure node scheduling: If you need to run a node periodically, configure its Scheduling Policy in the Scheduling Settings panel on the right.

  • Publish a node: To run a task in the production environment, click the image icon to publish the node. A node runs on schedule only after it is published to the production environment.

  • Task O&M: After a task is published, monitor its periodic runs in the Operation Center. For more information, see Get started with Operation Center.

Related documents