CDH Spark node-DataWorks(DataWorks)-阿里云帮助中心

Spark is a general-purpose big data analytics engine known for its high performance, ease of use, and versatility. You can use Spark for complex in-memory analytics and to build large-scale, low-latency data analysis applications. DataWorks provides CDH Spark nodes to help you develop and periodically schedule Spark tasks. This topic describes how to configure and use a CDH Spark node.

Prerequisites

You have created an Alibaba Cloud CDH cluster and associated it with a DataWorks workspace. For more information, see New Data Development: Bind a CDH Compute Engine.

Important
You have installed the Spark component in the CDH cluster and configured Spark-related information when you bound the cluster.
(Optional, for RAM users) The RAM user for task development has been added to the corresponding workspace and granted the Development or Workspace Administrator role. The Workspace Administrator role has extensive permissions, so grant it with caution. For more information about adding members, see Add members to a workspace.

Note
If you use an Alibaba Cloud account, you can skip this step.
You have configured a Hive data source in DataWorks and verified that the data source passed the connectivity test. For more information, see Manage data sources.

Preparations: Develop the task and get the JAR package

Before you use DataWorks to schedule a CDH Spark task, you must develop the Spark task code in your CDH environment, compile the code, and generate a JAR package. For guidance on developing CDH Spark tasks, see Spark overview.

Create a CDH JAR resource

You can upload the task's JAR package to DataWorks. This allows DataWorks to periodically schedule the Spark task on your CDH cluster.

For more information, see resource management. You can upload the JAR package from your local machine to the resource storage directory. Click the Click Upload button to upload the JAR resource.
Select the Storage Path, Data Sources, and Resource Group.
Click Save.

Create a node

For instructions, see Create a node.

Develop the node

On the CDH Spark node edit page, perform the following operations.

Reference a CDH JAR resource

Open the CDH Spark node that you created and go to the code editor.
In the resource list in the left-side navigation pane, find the resource that you want to reference, right-click the resource, and select Insert Resource Path.
After you reference the resource, a ##@resource_reference{""} statement appears in the code editor, confirming the reference. Add your JAR package name to this statement. The file name in the following command is an example. Replace it with your actual information.

##@resource_reference{"spark_examples_2.11_2.4.0.jar"}
spark_examples_2.11_2.4.0.jar

Edit the CDH Spark node code

Modify the CDH Spark node code to add the spark-submit command. The following code shows an example:

Important

CDH Spark nodes do not support comment statements in the code editor. You must write the task code as shown in the following example. Do not add comments, or an error will occur when you run the node.

##@resource_reference{"spark-examples_2.11-2.4.0.jar"}
spark-submit --class org.apache.spark.examples.SparkPi --master yarn  spark-examples_2.11-2.4.0.jar 100

Note

org.apache.spark.examples.SparkPi: The main class of the task in your compiled JAR package.
spark-examples_2.11-2.4.0.jar: The name of the CDH JAR resource that you uploaded.

Debug the node

In the Run Configuration's Compute Resource section, select and configure the Compute Resource and Resource Group.
1. For Computing resource, select the name of the CDH cluster that you registered in DataWorks.
2. Select the scheduling resource group that passed the connectivity test with the data source. For more information, see Network Connectivity Solutions.
Note
You can also adjust the CUs for Scheduling based on the resources required for task execution. The default value is 0.5.
In the toolbar at the top of the node editor, click Run.

Next steps

Configure node scheduling: If you need to run a node periodically, configure its Scheduling Policy in the Scheduling Settings panel on the right.
Publish a node: To run a task in the production environment, click the icon to publish the node. A node runs on schedule only after it is published to the production environment.
Task O&M: After a task is published, you can monitor the status of its periodic runs in the Operation Center. For more information, see Get started with Operation Center.