DocsICP FilingConsole
Documentation
Home Page E-MapReduce EMR Serverless Spark User Guide Cluster management Use Spark clusters

Use Spark clusters

Updated at:
Copy as MD
Product
My Favorites

Spark clusters provide an always-on cluster mode in EMR Serverless Spark. Once started, a cluster keeps running and accepts task submissions immediately without waiting for resources to spin up, making it ideal for latency-sensitive workloads.

Use cases

Spark clusters are suited for:

  • Short-running queries that need low startup latency.

  • Workloads that submit batch tasks frequently.

If your tasks are infrequent and startup time is not a concern, use the standard Serverless mode to submit tasks on demand and avoid the cost of an idle cluster.

Limitations

  • Task concurrency: A single Spark cluster supports up to 200 concurrent tasks by default. To request a higher limit, submit a ticket.

  • Kyuubi/Livy Gateway: Tasks submitted through Kyuubi Gateway or Livy Gateway cannot be directed to a Spark cluster.

  • Authentication: LDAP and Kerberos authentication are not supported.

  • Authorization: Ranger authorization is not supported.

  • Managed file directory mount: Mounting an integrated file directory is not supported.

Billing

After a Spark cluster starts, you are billed continuously for the resources it occupies (Master and Worker nodes), regardless of whether any tasks are running. Tasks submitted to the cluster incur no additional charges because their resource consumption is covered by the cluster fee.

For pricing details, see Compute resources (pay-as-you-go).

Create a Spark cluster

  1. Log on to the E-MapReduce console.

  2. In the left-side navigation pane, choose EMR Serverless > Spark.

  3. Click the name of the target workspace to open the Spark console.

  4. In the left-side navigation pane, choose O&M Center > Cluster Management.

  5. On the Clusters page, click the Spark Cluster tab.

  6. Click Create Spark Cluster.

  7. Configure the cluster settings.

    Parameter

    Description

    Cluster Name

    A custom name for the Spark cluster.

    Engine Version

    The Spark engine version to use.

    Service High Availability

    Whether to enable service high availability. Disabled by default.

    Cluster Resource Configuration

    Configure Master and Worker nodes:
    - Resource Queue: Select the resource queue for deploying the nodes.
    - Specification: Select the node specification (vCPU, memory).
    - Number / Elastic configuration: Set the node count. Worker nodes support elastic scaling with a minimum and maximum node count.

    Network Connection

    The network connection for the cluster. To create a new one, go to the Network Connection page.

  8. Click Create Cache Cluster.

Submit tasks to a Spark cluster

Submit tasks to a Spark cluster in any of the following ways.

Note

  • When you submit a task to a Spark cluster through the console or workflow orchestration, the system automatically starts the cluster if it is not running.

  • When a task runs on a Spark cluster, the engine version, network connection, and integrated file directory mount settings are determined by the cluster configuration, not the task configuration.

Method 1: Submit from the Development page

  1. On the Development page, create a batch task of type SQL, JAR, or PySpark.

  2. Click the execution resource dropdown in the upper-right corner and select the target Spark cluster.

  3. Click Publish or Run.

Method 2: Submit via spark-submit, API, or DataWorks

Add the following Spark configuration to your submission parameters to direct the task to a specific Spark cluster:

spark.emr.serverless.standalone.clusterId <your-Spark-cluster-id>

Replace <your-Spark-cluster-id> with the actual Spark cluster ID. You can find the cluster ID on the cluster list page or the cluster details page.

Note

Note Tasks submitted from DataWorks through the Serverless Spark Kyuubi path cannot be directed to a Spark cluster.

Method 3: Submit a streaming task from the Workflows page

When starting a streaming task on the Workflows page, select a Spark cluster as the execution resource.

View task history

Tasks submitted to a Spark cluster are recorded in the task history. Whether submitted from the Development page, spark-submit, or the API, all tasks appear under O&M Center > Job History.

Resize the Worker node pool

If a Spark cluster uses elastic Worker configuration, you can change the minimum and maximum number of Worker nodes without restarting the cluster. The cluster ID stays the same and running tasks are not interrupted.

Before you begin, make sure that the cluster meets the following requirements:

  • The engine version is esr-4.9.1 (Spark 3.5.2, Scala 2.12)-standalone or later.

  • The cluster is running and is not applying a configuration update.

  1. Log on to the E-MapReduce console.

  2. In the left-side navigation pane, choose EMR Serverless > Spark.

  3. Click the name of the target workspace to open the Spark console.

  4. In the left-side navigation pane, choose O&M Center > Cluster Management.

  5. On the Clusters page, click the Spark Cluster tab.

  6. Find the target cluster and choose More > Adjust Worker Count in the Actions column.

  7. Set the minimum and maximum number of Worker nodes.

  8. Click Save, and then click OK in the confirmation dialog box.

Note

The new configuration takes effect immediately. The cluster briefly shows Running (Updating configuration) and returns to Running after the Worker node count falls within the new range.

FAQ

What is the difference between a Spark cluster and Serverless mode?

Aspect

Serverless mode

Spark cluster mode

Resource model

Resources spin up on demand and are released after use.

An always-on cluster that keeps running once started. You can stop or resize it at any time.

Startup speed

Tasks must wait for resources to spin up.

Tasks are submitted directly to a running cluster, resulting in faster startup.

Billing

Billed based on the resources each task actually uses.

Billed continuously based on the resources the cluster occupies.

Use cases

Infrequent tasks where startup time is not a concern.

Short queries, frequent submissions, and streaming tasks that need fast startup.

How do I choose a deploy queue when creating a Spark cluster?

Use a dedicated resource queue for the Spark cluster to avoid resource contention with other Serverless tasks. For testing purposes, you can use an existing queue and manually reduce the Worker node count.

What happens to tasks when a cluster stops?

When a cluster stops, any tasks that have been submitted but not yet finished fail with the message "Spark cluster stopped." You can view failure details in the task history.

What if the task concurrency limit is not enough?

The default task concurrency limit for a Spark cluster is 200. To increase it, submit a ticket to request custom cluster parameters and apply the following configuration:

master.SPARK_MASTER_OPTS   -Dspark.deploy.retainedApplications=200 -Dspark.deploy.retainedDrivers=200

How do I use a runtime environment in a Spark cluster?

Submit a ticket to request custom cluster parameters, then configure the runtime environment as follows:

  • When creating the cluster, add the following parameter in the cluster advanced configuration:

    spark.standalone.pyenv.ids   <runtime-environment-id>

  • When submitting a task, select the runtime environment to use for that task.

Previous：Use a custom imageNext：AI Center