Use DataWorks Spark

Updated at:

This topic describes how to create and configure a MaxCompute Spark node in DataWorks (Data Governance Platform), including how to upload resources, configure the node, and pass parameters.

Procedure

Step 1: Create a resource

  1. Log in to the DataWorks console and select a region in the upper-left corner.

  2. Select Workspace, and click Go to Data Studio.

  3. In the leftmost navigation pane, click the image **Resource Management** icon. After you create a directory, click the image **Create Resource** icon next to the directory and select **MaxCompute Jar**. Upload the JAR file that is compiled from the template project.

  4. After uploading the resource, click **Publish Resource**. Otherwise, the Spark node cannot recognize the resource.

Step 2: Create a MaxCompute Spark node

  1. In the leftmost navigation pane, click the image **Data Development** icon. Click the image **Create Node** icon and select MaxCompute > MaxCompute Spark.

  2. Select the jar resource that you uploaded in the previous step and fill in the configuration fields on the DataWorks page based on the settings in spark-defaults.conf. The Main Class and Parameters fields specify the main class and its arguments. Click Publish.

  3. After publishing the node, click Go to Operation Center. On the new page, click Test in the Actions column for the new node.

DataWorks Spark node configuration

Overview

The configuration of a MaxCompute Spark node in DataWorks essentially corresponds to the parameters and options of the spark-submit command. The following table shows the specific mapping:

DataWorks UI field

spark-submit parameter

Main Java/Python resource

app jar or python file

Configurations

--conf PROP=VALUE

Main Class

--class CLASS_NAME

Parameters

[app arguments]

Select JAR resources

--jars JARS

Select Python resources

--py-files PY_FILES

Select file resources

--files FILES

Select archive resources

--archives ARCHIVES

Configurations

This field corresponds to the --conf option of the spark-submit command, as shown in the second row of the preceding table.

  • You do not need to configure accessid, accesskey, projectname, and endpoint because they default to the production account (you can explicitly configure them to override the default values).

  • You must add each configuration from the spark-default.conf file to the Configurations field in DataWorks.

Pass parameters

You can pass parameters, such as bizdate, to DataWorks Spark nodes. The steps are as follows:

  1. In the Scheduling > Parameters section, add the parameter in the same way as for an SQL node.

  2. In the Parameters field of the Spark node, reference the parameter. DataWorks passes this parameter to your main class, where you can parse it in your code.

Parameter retrieval methods vary by language:

  • Java/Scala: Retrieve parameters through the args array of the main class.

  • Python: Read parameters from sys.argv.

Upload resources

You can add the resources that your job requires in DataWorks. At runtime, DataWorks uploads these resources to the working directory. Resources can include the following:

  • jar resource/Python resource: These map to the --jars and --py-files parameters of the spark-submit command.

  • file resource: This maps to the --files parameter of the spark-submit command.

  • archive resource: This maps to the --archives parameter of the spark-submit command. DataWorks extracts archive resources by default. The name of the resulting folder is the resource name without the file extension. For example, if you upload a resource named mnist.zip, the extracted folder is named mnist.

The maximum size for a resource uploaded to DataWorks is 50 MB. To use a larger resource, you must upload it as a MaxCompute resource by using the MaxCompute client (odpscmd) and then add it as a resource in Data Development.