DocsICP FilingConsole
官方文档
首页

Serverless Spark Batch node

更新时间：
复制 MD 格式
产品详情
我的收藏

Develop and periodically schedule Spark jobs on an EMR Serverless Spark cluster in DataWorks. Spark supports complex in-memory computing, making it well suited for large-scale, low-latency data analytics.

Prerequisites

  • Compute resource limitations: Only bound EMR Serverless Spark compute resources are supported. You must ensure that the resource group and the compute resource can communicate over the network.

  • Resource group constraints: This task runs only in a Serverless resource group.

  • (Optional, for RAM users) The Resource Access Management (RAM) user for task development must be added to the workspace and assigned the Development or Workspace Administrator role (this role includes extensive permissions and must be granted with caution). For more information, see Add workspace members.

    If you are using a root account, skip this step.

Create a node

For instructions, see Create a node.

Develop the node

Note

Before you develop a Serverless Spark Batch job, develop the Spark job code in E-MapReduce (EMR) and compile it into a JAR package. For guidance on Spark development, see Spark tutorials.

Choose the approach that best fits your use case:

Option 1: Upload and reference EMR JAR

Upload the compiled JAR package to DataWorks as an EMR JAR resource. This approach works for JAR packages smaller than 500 MB.

  1. Create an EMR JAR resource.

    1. In the left-side navigation pane, click the image icon to go to the Resource Management page.

    2. On the Resource Management page, click the image icon, select New Resource, and name the resource spark-examples_2.11-2.4.0.jar.

    3. Click Click Upload and upload the spark-examples_2.11-2.4.0.jar file.

    4. Select a Storage Path, Data Sources, and Resource Group.

      Important

      For the data source, you must select the bound EMR Serverless Spark cluster.

    5. Click Save.

  2. Reference the EMR JAR resource.

    1. Open the Serverless Spark Batch node that you created and stay on the code editor page.

    2. In the Resource Management pane on the left, find the resource you want to use, right-click it, and select Insert Resource Path.

    3. The resource is successfully referenced when the following code is automatically added to the code editor of the Serverless Spark Batch node:

      ##@resource_reference{"spark-examples_2.11-2.4.0.jar"}
spark-examples_2.11-2.4.0.jar

      In the preceding code, spark-examples_2.11-2.4.0.jar is the name of the EMR JAR resource that you uploaded.

    4. Modify the node code by adding the spark-submit command. The following example shows the modified code.

      Important

      • Follow the example exactly. Do not add comments, as they cause the node to fail.

      • For EMR Serverless Spark, you do not need to set the deploy-mode parameter for the spark-submit command. Only cluster mode is supported.

      ##@resource_reference{"spark-examples_2.11-2.4.0.jar"}
spark-submit --class org.apache.spark.examples.SparkPi spark-examples_2.11-2.4.0.jar 100

      Command

      Description

      class

      The main class of the application in the JAR file. In this example, the value is org.apache.spark.examples.SparkPi.
      Note

      For more information about parameters, see Submit a job using spark-submit.

Option 2: Reference OSS resource

Reference a resource stored in Object Storage Service (OSS). DataWorks automatically loads the specified OSS resource at runtime. Use this approach when an EMR job requires JAR dependencies or depends on scripts.

  1. Develop the JAR resource: This topic uses SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar as an example.

  2. Upload the JAR resource.

    1. Log on to the OSS console. In the left-side navigation pane, click buckets.

    2. Click the name of the target bucket to go to the file management page.

    3. Click Create Directory to create a directory to store the JAR resource.

    4. Go to the directory and upload the SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar file to the bucket.

  3. Reference the JAR resource.

    1. In the code editor for the Serverless Spark Batch node that you created, write the code to reference the JAR resource.

      Important

      In the following code, the OSS bucket name is mybucket and the directory is emr. Replace them with your actual values.

      spark-submit --class com.aliyun.emr.example.spark.SparkMaxComputeDemo oss://mybucket/emr/SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar

      Parameters:

      Parameter

      Description

      class

      The fully qualified name of the main class to run.

      oss file path

      The format is oss://{bucket}/{object}.

      • bucket: The OSS container that stores objects. Each bucket has a unique name. You can log on to the OSS console to view all buckets under your account.

      • object: A file or path stored in a bucket.
      Note

      For more information about parameters, see Submit a job using spark-submit.

Debug the node

  1. In the Run Configuration pane, configure settings such as Compute resource and resource group.

    Parameter

    Description

    Compute resource

    Select a bound EMR Serverless Spark compute resource. If none are available, select Create Compute Resource from the drop-down list.

    resource group

    Select a resource group that is bound to the workspace.

    Script parameters

    Define variables in the node's code by using the ${Parameter name} format. Then define the Parameter name and Parameter Value in the Script Parameters section. At runtime, DataWorks replaces these variables with the specified values. For more information, see Scheduling parameters.

    ServerlessSpark node parameters

    Runtime parameters for the Spark program. The following types are supported:

    The configuration format is: "spark.eventLog.enabled": false. DataWorks automatically adds this setting to the code submitted to Serverless Spark in the --conf key=value format.

    Note

    You can set global Spark parameters at the workspace level and specify whether they take priority over parameters set within a specific module. For details, see Configure global Spark parameters.

  2. In the toolbar at the top of the node editor, click Run.

    Important

    Before you publish, you must synchronize the ServerlessSpark node parameters in the Run Configuration to the ServerlessSpark node parameters in the Scheduling Settings.

Next steps

  • Configure node scheduling: If you need to run a node periodically, configure its Scheduling Policy in the Scheduling Settings panel on the right.

  • Publish a node: To run a task in the production environment, click the image icon to publish the node. A node runs on schedule only after it is published to the production environment.

  • Task O&M: After a task is published, you can monitor the status of its periodic runs in the Operation Center. For more information, see Get started with Operation Center.

Related documentation

Appendix: DataWorks parameters

Parameter

Description

SERVERLESS_QUEUE_NAME

The resource queue to which the job is submitted. By default, jobs are submitted to the Default Resource Queue configured for the cluster under Clusters in the Management Center. To isolate and manage resources, you can add queues. For more information, see Manage resource queues.

Configuration methods:

  • Specify the resource queue for the job by using node parameters.

  • Specify the resource queue for the job by using global Spark parameters.
Previous：Serverless SparkNext：Serverless Spark SQL node