Create a CDH MR node-DataWorks(DataWorks)-阿里云帮助中心

Prerequisites

Create a workflow in DataStudio.

DataStudio uses workflows to organize development work across different compute engines. Therefore, you must create a workflow before you can create a node. For more information, see Create a workflow.
Create a CDH cluster and register it with DataWorks.

Before you create and develop CDH-related nodes and tasks, you must register the CDH cluster with your DataWorks workspace. For more information, see DataStudio (old version): Bind a CDH computing resource.
(Optional) If you are using a RAM user, add the user to the workspace and grant them either the Development or Workspace Administrator role. The Workspace Administrator role grants extensive permissions, so assign it with caution. For more information, see Add members to a workspace and assign roles to them.
Purchase and configure a serverless resource group. This includes binding the resource group to a workspace and setting up network connections. For more information, see Use a serverless resource group.

This task type can run on a serverless resource group (recommended) or an older exclusive resource group for scheduling.

Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
Right-click a workflow and choose Create Node > cdh > CDH MR.
In the Create Node dialog box, configure the node's engine instance, path, and name.
Click OK. The node editor opens, where you can develop and configure the task.

DataWorks allows you to upload resources from your local machine to DataStudio and then reference them in your nodes.

Create a CDH JAR resource.

In the corresponding workflow, right-click cdh > Resources and choose Create Resource > CDH JAR. In the Create Resource dialog box, click Click Upload and select the file that you want to upload.

In the dialog box, the Storage path defaults to /user/admin/lib. If Kerberos authentication is enabled, you must grant the current user write permissions on this directory. The uploaded JAR package cannot exceed 50 MB in size, and its Name must end with the .jar suffix.
Reference the CDH JAR resource.
1. Open the created CDH MR node to its edit page.
2. Under cdh > Resources, find the resource that you want to reference (for example, onaliyun_mr_wordcount-1.0-SNAPSHOT.jar). Right-click the resource name and choose Insert Resource Path.
  
  Selecting Insert Resource Path adds a statement in the ##@resource_reference{""} format to the code editor, which indicates that the resource is successfully referenced. The following command is an example. Replace the placeholder resource package, bucket name, and path with your actual information.
```
##@resource_reference{"onaliyun_mr_wordcount-1.0-SNAPSHOT.jar"}
onaliyun_mr_wordcount-1.0-SNAPSHOT.jar cn.apache.hadoop.onaliyun.examples.EmrWordCount oss://onaliyun-bucket-2/cdh/datas/wordcount02/inputs oss://onaliyun-bucket-2/cdh/datas/wordcount02/outputs
```
  Note
  Comments are not supported in the code editor for CDH MR nodes.

To run the node task on a schedule, click Scheduling on the right side of the node configuration tab and configure the scheduling properties:

Configure the basic properties of the task. For more information, see Configure basic properties.
Configure the scheduling cycle, rerun properties, and dependencies. For more information, see Configure time properties and Configure same-cycle scheduling dependencies.

Note
You must set the Rerun attribute and Parent Nodes properties for the node before you can commit it.
Configure resource properties. For more information, see Configure resource properties. When accessing the internet or a virtual private cloud (VPC), select a resource group for scheduling that provides network connectivity to the target node. For more information, see Network connectivity solutions.

(Optional) Select a resource group and assign values to custom parameters.
- On the toolbar, click the icon. In the Parameter dialog box, select the resource group to use for debugging.
- If your code uses scheduling parameters, assign test values to them here for debugging. For more information about the parameter assignment logic, see What are the differences in value assignment logic for Run, Advanced Run, and smoke testing in the development environment?.
Save and run the SQL statements.

In the toolbar, click the (Save) icon to save the statements. Then, click the (Run) icon to run the task.
(Optional) Perform smoke testing.

You can perform smoke testing in the development environment either during or after committing the node. For more information, see Perform smoke testing.

Submit and deploy the node task.
1. Click the (Save) icon in the toolbar to save the node.
2. Click the (Submit) icon in the toolbar to submit the node task.
3. In the Commit Node dialog box, enter a Change Description.
4. Click Determine.
If you use a standard-mode workspace, you must publish the submitted task to the production environment. To do so, click Task Publishing on the left side of the top menu bar. For more information, see Publish tasks.
View the periodically scheduled task.
1. In the upper-right corner of the editor, click O&M Personnel to go to Operation Center in the production environment.
2. View the running periodically scheduled task. For more information, see Manage periodic tasks.
To view more details about periodically scheduled tasks, click Operation Center in the top menu bar. For more information, see Operation Center overview.