Use DataWorks Spark
This topic describes how to create and configure a MaxCompute Spark node in DataWorks (Data Governance Platform), including how to upload resources, configure the node, and pass parameters.
Procedure
Step 1: Create a resource
-
Log in to the DataWorks console and select a region in the upper-left corner.
-
Select Workspace, and click Go to Data Studio.
-
In the leftmost navigation pane, click the
**Resource Management** icon. After you create a directory, click the
**Create Resource** icon next to the directory and select **MaxCompute Jar**. Upload the JAR file that is compiled from the template project. -
After uploading the resource, click **Publish Resource**. Otherwise, the Spark node cannot recognize the resource.
Step 2: Create a MaxCompute Spark node
-
In the leftmost navigation pane, click the
**Data Development** icon. Click the
**Create Node** icon and select . -
Select the jar resource that you uploaded in the previous step and fill in the configuration fields on the DataWorks page based on the settings in
spark-defaults.conf. The Main Class and Parameters fields specify the main class and its arguments. Click Publish. -
After publishing the node, click Go to Operation Center. On the new page, click Test in the Actions column for the new node.
DataWorks Spark node configuration
Overview
The configuration of a MaxCompute Spark node in DataWorks essentially corresponds to the parameters and options of the spark-submit command. The following table shows the specific mapping:
|
DataWorks UI field |
spark-submit parameter |
|
Main Java/Python resource |
|
|
Configurations |
|
|
Main Class |
|
|
Parameters |
|
|
Select JAR resources |
|
|
Select Python resources |
|
|
Select file resources |
|
|
Select archive resources |
|
Configurations
This field corresponds to the --conf option of the spark-submit command, as shown in the second row of the preceding table.
-
You do not need to configure
accessid,accesskey,projectname, andendpointbecause they default to the production account (you can explicitly configure them to override the default values). -
You must add each configuration from the
spark-default.conffile to the Configurations field in DataWorks.
Pass parameters
You can pass parameters, such as bizdate, to DataWorks Spark nodes. The steps are as follows:
-
In the Scheduling > Parameters section, add the parameter in the same way as for an SQL node.
-
In the Parameters field of the Spark node, reference the parameter. DataWorks passes this parameter to your main class, where you can parse it in your code.
Parameter retrieval methods vary by language:
-
Java/Scala: Retrieve parameters through the
argsarray of the main class. -
Python: Read parameters from
sys.argv.
Upload resources
You can add the resources that your job requires in DataWorks. At runtime, DataWorks uploads these resources to the working directory. Resources can include the following:
-
jar resource/Python resource: These map to the
--jarsand--py-filesparameters of thespark-submitcommand. -
file resource: This maps to the
--filesparameter of thespark-submitcommand. -
archive resource: This maps to the
--archivesparameter of thespark-submitcommand. DataWorks extracts archive resources by default. The name of the resulting folder is the resource name without the file extension. For example, if you upload a resource namedmnist.zip, the extracted folder is namedmnist.
The maximum size for a resource uploaded to DataWorks is 50 MB. To use a larger resource, you must upload it as a MaxCompute resource by using the MaxCompute client (odpscmd) and then add it as a resource in Data Development.