DocsICP FilingConsole
官方文档
首页

Serverless Spark Batch node

更新时间：
复制 MD 格式
产品详情
我的收藏

Spark is a general-purpose engine for large-scale data analytics, known for its high performance, ease of use, and broad applicability. It supports complex in-memory computing, which is ideal for building large-scale, low-latency data applications. DataWorks lets you use Serverless Spark Batch nodes to develop and periodically schedule Spark jobs on an EMR Serverless Spark cluster in DataWorks.

Limitations

  • Compute resource limits: Only an EMR Serverless Spark compute resource is supported. Ensure that the resource group and the compute resource are connected over the network.

  • Resource group constraints: This task runs only in a Serverless resource group.

  • (Optional, for RAM users) The RAM user for task development has been added to the corresponding workspace and granted the Development or Workspace Administrator (this role has extensive permissions, grant with caution) role. For more information about how to add members, see Add members to a workspace.

    If you use an Alibaba Cloud account, you can skip this step.

Create a node

For instructions, see Create a node.

Develop the node

Note

Before you develop a Serverless Spark Batch job, first develop your Spark job code in EMR and compile it into a JAR package. For guidance on Spark development, see Spark Tutorial.

Choose the method that best fits your scenario:

Reference a DataWorks resource

In DataWorks, you can upload resources from your local machine to Data Studio and then reference them. After your Serverless Spark Batch job is compiled, obtain the resulting JAR package. We recommend that you choose a storage method based on the size of the JAR package. If the package is smaller than 500 MB, you can upload it from your local machine as a DataWorks EMR JAR resource.

  1. Create an EMR JAR resource.

    1. In the left-side navigation pane, click the image icon to go to the Resource Management page.

    2. On the Resource Management page, click the image icon, select New Resource, and enter a name, such as spark-examples_2.11-2.4.0.jar.

    3. Click Click Upload to upload spark-examples_2.11-2.4.0.jar.

    4. Select a Storage Path, Data Sources, and Resource Group.

      Important

      For the Data Source, select the bound EMR Serverless Spark cluster.

    5. Click Save.

  2. Reference the EMR JAR resource.

    1. Open the Serverless Spark Batch node that you created and stay on the code editor page.

    2. In the left-side navigation pane, find the resource that you want to reference under Resource Management. Right-click the resource and select Insert Resource Path.

    3. When you select this option, a reference statement is automatically added to the code editor of the Serverless Spark Batch node, confirming that the resource is successfully referenced.

      ##@resource_reference{"spark-examples_2.11-2.4.0.jar"}
spark-examples_2.11-2.4.0.jar

      In this code, spark-examples_2.11-2.4.0.jar is the name of the EMR JAR resource that you uploaded.

    4. Add the spark-submit command to the Serverless Spark Batch node code. The following code is an example.

      Important

      • The code editor for Serverless Spark Batch nodes does not support comments. To prevent runtime errors, remove all comments from your code.

      • You do not need to set the deploy-mode parameter for the spark-submit command because EMR Serverless Spark supports only cluster mode.

      ##@resource_reference{"spark-examples_2.11-2.4.0.jar"}
spark-submit --class org.apache.spark.examples.SparkPi spark-examples_2.11-2.4.0.jar 100

      Parameter

      Description

      class

      The main class of the job in the compiled JAR package. In this example, the main class is org.apache.spark.examples.SparkPi.
      Note

      For more information about the parameters, see Submit a job by using spark-submit.

Reference an OSS resource

You can directly reference a resource from OSS. When an EMR node runs, DataWorks automatically loads the OSS resources referenced in your code to a local path. This method is commonly used in scenarios where an EMR job requires JAR dependencies or depends on scripts.

  1. Develop a JAR resource: This topic uses SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar as an example.

  2. Upload the JAR resource.

    1. Log on to the OSS console and click Buckets in the left-side navigation pane of the region where your bucket resides.

    2. Click the name of the destination bucket to open its file management page.

    3. Click Create Directory to create a directory to store the JAR resource.

    4. Go to the directory and upload the SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar file to the bucket.

  3. Reference the JAR resource.

    1. On the code editor page of the Serverless Spark Batch node, edit the code to reference the JAR resource.

      Important

      In the following code, the OSS bucket name is mybucket and the directory is emr. Replace them with your actual bucket name and path.

      spark-submit --class com.aliyun.emr.example.spark.SparkMaxComputeDemo oss://mybucket/emr/SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar

      The following table describes the reference parameters.

      Parameter

      Description

      class

      The fully qualified name of the main class to run.

      oss file path

      The format is oss://{bucket}/{object}.

      • bucket: A container for storing objects in OSS. Each bucket has a unique name. You can log on to the OSS console to view all buckets under your account.

      • object: a specific object, such as a file name or path, stored in a bucket.
      Note

      For more information about the parameters, see Submit a job by using spark-submit.

Debug the node

  1. In the Run Configuration section, configure parameters such as Compute Resource and Resource Group.

    Parameter

    Description

    Compute resource

    Select a bound EMR Serverless Spark compute resource. If no compute resources are available, you can select Create Compute Resource from the drop-down list.

    Resource group

    Select a resource group that is bound to the workspace.

    Script parameters

    When you configure the node, you can define variables in the ${ParameterName} format. Then, you must configure the Script Parameters and Parameter name in the Parameter Value section. At runtime, the variables are dynamically replaced with their actual values. For more information, see Sources and expressions of scheduling parameters.

    ServerlessSpark node parameters

    Runtime parameters for the Spark program. Supported parameters include:

    Configure the parameters in the following format: "spark.eventLog.enabled": false. DataWorks automatically adds these parameters to the final code submitted to EMR Serverless Spark in the --conf key=value format.

    Note

    DataWorks allows you to configure global Spark parameters. These parameters are specified at the workspace level and apply to all DataWorks modules. You can specify whether the priority of these global Spark parameters is higher than that of the Spark parameters within a specific module. For more information about how to configure global Spark parameters, see Configure global Spark parameters.

  2. In the toolbar at the top of the node editor page, click Run to run the job.

    Important

    Before you publish the node, synchronize the Run Configuration from the Run Configuration section to the Scheduling Settings field in the Scheduling section.

Next steps

  • Configure node scheduling: If you need to run a node periodically, configure its Scheduling Policy in the Scheduling Settings panel on the right.

  • Publish a node: To run a task in the production environment, click the image icon to publish the node. A node runs on schedule only after it is published to the production environment.

  • Task O&M: After a task is published, you can monitor the status of its periodic runs in the Operation Center. For more information, see Get started with Operation Center.

Related documentation

Appendix: DataWorks parameters

Parameter

Description

SERVERLESS_QUEUE_NAME

Specifies the resource queue where the job is submitted. By default, jobs are submitted to the Default Resource Queue configured for the cluster in Clusters of Management Center. You can add queues to implement resource isolation and management. For more information, see Manage resource queues.

You can configure this parameter in one of the following ways:
Previous：Serverless Kyuubi nodeNext：Serverless Spark SQL node