EMR PySpark node
DataWorks provides EMR PySpark nodes, where you can collaboratively write Python business logic and spark-submit submission commands in a unified development interface. The nodes connect to Alibaba Cloud EMR semi-managed clusters and fully managed EMR Serverless Spark.
Scope
Compute resource limits:EMR PySpark nodes support these two compute resource types:
EMR compute resources (semi-managed clusters): only DataLake clusters and Custom clusters.
EMR Serverless Spark compute resources (fully managed clusters).
ImportantTo run an EMR PySpark node on an EMR semi-managed cluster, such as a DataLake or Custom cluster, submit a ticket. Include your cluster type, EMR version, and intended use case so that we can assess integration feasibility and provide support.
Resource group limits:Only Serverless resource groups are supported.
Permission limits:The Alibaba Cloud account owner or a RAM user who has been added to the workspace and has the Dev or Workspace Administrator role. For instructions on adding members, see Add members to a workspace are supported.
Create a node
Log on to the DataWorks console. Switch to the target region, then click in the left navigation pane. Select the workspace from the drop-down list and click Data Analytics are supported.
On the Data Development (Data Studio) page, create an EMR PySpark node.
Configure the nodePath and Name. In this example, the node is named emr_pyspark_test.
Develop the node
An EMR PySpark node uses a two-pane collaborative editor:
Upper pane for Python code. Write the core business logic here. You can reference uploaded resource files, such as
.pymodules.Lower pane for Submit commands. Use it to submit spark-submit commands to the EMR cluster.
The following Monte Carlo example shows how an EMR PySpark node estimates pi in parallel.
Step 1: Upload dependency resources
Upload your custom Python file (in this example, utils.py) to the DataWorks resource management module. After the upload, you can reference this resource in your node. This Python file defines the core computation logic for Monte Carlo sampling within a single Spark task.
For more information about how to upload resources, see Create and manage resources. On the resource management page in Data Studio, click Create Resource, select the resource type as
EMR File, and configure the resource Name.Click Re-Upload to upload the utils.py file used in this example.
Select the Storage path, Connection, and Resource Groups, and then click Save.
Step 2: Write Python code
In the Python code editor, enter the following code. The example estimates pi by dividing the total work into tasks and distributing them across cluster compute nodes. After parallel execution, it collects and aggregates the results to produce the final estimate.
##@resource_reference{"utils.py"}
from pyspark.sql import SparkSession
from utils import estimate_pi_in_task
import sys
def main():
# Create a SparkSession
spark = SparkSession.builder \
.appName("EstimatePi") \
.getOrCreate()
sc = spark.sparkContext
# Total number of samples
total_samples = int(sys.argv[1])
num_partitions = ${test1}
# Samples per partition
samples_per_partition = total_samples // num_partitions
# Create an RDD and run estimate_pi_in_task once per partition
rdd = sc.parallelize(range(num_partitions), num_partitions)
# Map each partition to a sampling task
inside_counts = rdd.map(lambda _: estimate_pi_in_task(samples_per_partition))
# Aggregate results from all partitions
total_inside = inside_counts.sum()
pi_estimate = 4.0 * total_inside / total_samples
print(f"Total samples: {total_samples}")
print(f"Samples inside the circle: {total_inside}")
print(f"Estimated pi: {pi_estimate:.6f}")
spark.stop()
if __name__ == "__main__":
main()You can submit only the entire Python file as a Spark job; you cannot run a selected portion of the code.
Additional information about the script parameters:
Parameter | Type | Value | Description |
| Command-line argument |
| Specifies the total number of samples, for example |
| Scheduling parameter | Dynamically replaced with its actual value by DataWorks during execution or scheduling | Specifies the number of partitions, for example |
Step 3: Write the spark-submit command
In the Submit command editor, enter the following code. The command uses the spark-submit tool to package the specified Python application and submit it to an Alibaba Cloud EMR cluster.
The parameters supported by spark-submit differ significantly between EMR compute resources (semi-managed clusters) and EMR Serverless Spark compute resources (fully managed clusters). The following table compares sample code, command integration, and reference documents to help you configure them correctly.
Comparison item | EMR compute resources (semi-managed clusters) | EMR Serverless Spark compute resources (fully managed clusters) |
Sample code | | |
Command integration | Based on the | Based on the Alibaba Cloud EMR Serverless |
Command reference | For more information about parameters supported by | For parameter syntax, common options, and best practices for |
EMR PySpark nodes do not provide a dedicated Spark parameter configuration section. Custom Spark Conf parameters must be specified in the spark-submit command in the Submit Command pane using the --conf key=value format.
The following template parameter and engine version priorities apply only to EMR Serverless Spark compute resources, not to EMR semi-managed DataLake or Custom clusters; parameters and versions for those clusters are determined by the cluster.
When a node has multiple configuration sources, such as global Spark parameters and task-level parameters configured in the spark-submit command using --conf, Spark properties and engine versions take effect in the following priority orders (from highest to lowest):
Spark properties:
Parameters in the specified Spark configuration template referenced by
spark.emr.serverless.templateId.DataWorks global Spark parameters (effective when Global configuration takes priority is selected in Configure global Spark parameters).
Parameters specified in the current node's spark-submit command in the Submit Command pane in the
--conf key=valueformat.Parameters in the default Spark configuration template of EMR Serverless Spark (included in the priority order only when this default template is actually applied).
Engine versions:
The engine version specified by
--release-version "esr-4.1.1 (Spark 3.5.2, Scala 2.12)", for example, in the current node's spark-submit command in the Submit Command pane.The engine version specified in the Spark configuration template referenced by
spark.emr.serverless.templateId.The engine version specified in the default Spark configuration template of the EMR Serverless Spark workspace (included in the priority order only when this default template is actually applied).
The engine version specified by the task-level DataWorks parameter
SERVERLESS_RELEASE_VERSION, if configured.The default engine version of the EMR Serverless Spark compute resource bound to DataWorks.
Run the node
In Running Configurations, configure Compute Resource and Resource Group are supported.
Setting
Description
Compute resource
Select a bound EMR compute resource or or EMR Serverless Spark compute resource. If no compute resource is available, select Create Compute Resource are supported.
Resource group
Select a resource group bound to the workspace.
Script parameters
When configuring the node, define a variable in the form
${parameter_name}. Configure its Script Parameters information in Value Used in This Run. The task dynamically substitutes the actual value during execution.The parameter value is synchronized from the scheduling configuration.
NoteThe value for this run applies only to this execution. The system first uses Value Used in This Run; if unavailable, it uses Parameter Value are supported.
On the toolbar at the top of the node editor, click Run. The system combines the complete Python script, including resource references, and submits it to the EMR cluster through
spark-submit. It then returns execution logs and results.
Next steps
-
Configure node scheduling: If you need to run a node periodically, configure its Scheduling Policy in the Scheduling Settings panel on the right.
-
Publish a node: To run a task in the production environment, click the
icon to publish the node. A node runs on schedule only after it is published to the production environment. -
Task O&M: After a task is published, monitor its periodic runs in the Operation Center. For more information, see Get started with Operation Center.