How to create a CDH Spark node-DataWorks(DataWorks)-阿里云帮助中心

Prerequisites

Create a workflow in DataStudio.

DataStudio uses workflows to organize development work across different compute engines. Therefore, you must create a workflow before you can create a node. For more information, see Create a workflow.
Create a CDH cluster and register it with DataWorks.

Before you create and develop CDH-related nodes and tasks, you must register the CDH cluster with your DataWorks workspace. For more information, see DataStudio (old version): Bind a CDH computing resource.
(Optional) If you are using a RAM user, add the user to the workspace and grant them either the Development or Workspace Administrator role. The Workspace Administrator role grants extensive permissions, so assign it with caution. For more information, see Add members to a workspace and assign roles to them.
Purchase and configure a serverless resource group. This includes binding the resource group to a workspace and setting up network connections. For more information, see Use a serverless resource group.

Limitations

This type of task can run on a serverless resource group (recommended) or an exclusive resource group.

Develop the Spark task and JAR file

Before you schedule a CDH Spark task in DataWorks, you must develop the Spark task code in your CDH cluster, compile the code, and generate a JAR file. For more information about how to develop CDH Spark tasks, see Spark overview.

Note

You must upload the JAR file to DataWorks to periodically schedule the CDH Spark task.

Step 1: Create a CDH Spark node

Go to the DataStudio page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Development and O&M > Data Development. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.
Right-click a workflow and choose Create Node > cdh > CDH Spark.
In the Create Node dialog box, configure the engine instance, path, and name for the node.
Click OK. The node is created, and you can now proceed with task development and configuration.

Step 2: Create and reference a CDH JAR resource

Reference a JAR file in the CDH Spark node and write a spark-submit command to submit the task. Follow these steps:

Create a CDH JAR resource.

In the workflow, right-click cdh > Resources and choose Create Resource > CDH JAR. In the Create Resource dialog box, click Click Upload and select the JAR file to upload.
Reference the CDH JAR resource.
1. Open the CDH Spark node you created.
2. Under cdh > Resources, find the resource you want to use (for this example, spark-examples_2.11-2.4.0.jar), right-click its name, and select Insert Resource Path.
  
  After you select Insert Resource Path, a statement in the format ##@resource_reference{""} appears in the code editor. This indicates that you have successfully referenced the resource. The following example shows the result:
```
##@resource_reference{"spark-examples_2.11-2.4.0.jar"}
spark-examples_2.11-2.4.0.jar
```
3. Edit the CDH Spark node code to add the spark-submit command. The following example shows the modified code.
  
  Important
  Format your task code exactly as shown in the example. Adding comments will cause a runtime error.
```
##@resource_reference{"spark-examples_2.11-2.4.0.jar"}
spark-submit --class org.apache.spark.examples.SparkPi --master yarn  spark-examples_2.11-2.4.0.jar 100
```
  Parameters:
  - org.apache.spark.examples.SparkPi: The main class of the task in your compiled JAR file.
  - spark-examples_2.11-2.4.0.jar: The name of the CDH JAR resource that you uploaded.

Step 3: Configure task scheduling

To run the node task on a schedule, click Scheduling on the right side of the node configuration tab and configure the scheduling properties:

Configure the basic properties of the task. For more information, see Configure basic properties.
Configure the scheduling cycle, rerun properties, and dependencies. For more information, see Configure time properties and Configure same-cycle scheduling dependencies.

Note
You must set the Rerun attribute and Parent Nodes properties for the node before you can commit it.
Configure resource properties. For more information, see Configure resource properties. When accessing the internet or a virtual private cloud (VPC), select a resource group for scheduling that provides network connectivity to the target node. For more information, see Network connectivity solutions.

Step 4: Debug the task code

(Optional) Select a resource group and assign values to custom parameters.
- On the toolbar, click the icon. In the Parameter dialog box, select the resource group to use for debugging.
- If your code uses scheduling parameters, assign test values to them here for debugging. For more information about the parameter assignment logic, see What are the differences in value assignment logic for Run, Advanced Run, and smoke testing in the development environment?.
Save and run the SQL statements.

In the toolbar, click the icon to save the SQL statements, and then click the icon to run the task.
(Optional) Perform smoke testing.

You can perform smoke testing in the development environment either during or after committing the node. For more information, see Perform smoke testing.

Next steps

Commit and deploy the node task.
1. Click the Save icon in the toolbar to save the node.
2. Click the Commit icon in the toolbar to commit the node task.
3. In the Commit Node dialog box, enter a Change Description.
4. Click Determine.
If your workspace is in standard mode, you must deploy the task to the production environment after you commit it. On the left side of the top menu bar, click Deploy. For more information, see Deploy tasks.
View the scheduled tasks.
1. In the upper-right corner of the editor tab, click O&M Personnel to open the Operation Center page for the production environment.
2. View the status of the scheduled task. For more information, see Manage periodic tasks.
To view more details about scheduled tasks, you can click Operation Center in the top menu bar. For more information, see Operation Center overview.

Use cases

DataWorks provides comprehensive job scheduling and monitoring features to ensure that your Spark jobs are reliably submitted to your CDH cluster. This method simplifies O&M and improves resource management efficiency. The following are common use cases for Spark tasks:

Data analysis: Use Spark SQL, Dataset, and DataFrame APIs for complex data aggregation, filtering, and transformation to gain rapid insights from data.
Stream processing: Use Spark Streaming to process real-time data streams for instant analysis and decision-making.
Machine learning tasks: Use Spark MLlib for data preprocessing, feature extraction, and model training and evaluation.
Large-scale ETL tasks: Extract, transform, and load (ETL) large datasets to prepare data for a data warehouse or other storage systems.