CDH Spark SQL node-DataWorks(DataWorks)-阿里云帮助中心

Spark SQL is a distributed query engine that efficiently processes structured data. In DataWorks, you can use a CDH Spark SQL node to develop, schedule, and integrate CDH Spark SQL tasks with other jobs. This topic describes how to configure and use a CDH Spark SQL node.

Prerequisites

You have created an Alibaba Cloud CDH cluster and associated it with a DataWorks workspace. For more information, see New Data Development: Bind a CDH Compute Engine.

Important
You have installed the Spark component on the CDH cluster and configured Spark-related information during cluster binding.
(Optional, for RAM users) The RAM user for task development has been added to the corresponding workspace and granted the Development or Workspace Administrator role. The Workspace Administrator role has extensive permissions, so grant it with caution. For more information about adding members, see Add members to a workspace.

Note
If you use an Alibaba Cloud account, you can skip this step.
You have configured a Hive data source in DataWorks and verified that the data source passed the connectivity test. For more information, see Manage data sources.

Create a node

For instructions, see Create a node.

Develop the node

Write your task code in the SQL editor. You can define variables in your code in the ${variable_name} format, and then assign values to these variables in the Scheduling Parameters section of the Scheduling Settings panel. This allows you to dynamically pass parameters to the code during scheduled runs. For more information about scheduling parameters, see Sources and expressions of scheduling parameters. The following code provides an example:

CREATE TABLE IF NOT EXISTS test_spark.test_lineage_table_f1 (`id` BIGINT, `name` STRING)
PARTITIONED BY (`ds` STRING);
CREATE TABLE IF NOT EXISTS test_spark.test_lineage_table_t2 AS SELECT * FROM test_spark.test_lineage_table_f1;
INSERT INTO test_spark.test_lineage_table_t2 SELECT id,${var} FROM test_spark.test_lineage_table_f1;

Note

This example creates the test_lineage_table_f1 and test_lineage_table_t2 tables in the test_spark database and copies data from the test_lineage_table_f1 table to the test_lineage_table_t2 table. This example is for reference only. For actual use, replace the database environment with your own.
The ${var} parameter is set to the field name.

Debug the node

In the Compute Resource section of the Run Configuration, configure the Compute Resource and Resource Group.
1. For Compute Resource, select the name of the CDH cluster that you registered in DataWorks.
2. For Resource group, select a scheduling resource group that passed the connectivity test to the data source. For more information, see Network Connectivity Solution.
On the toolbar at the top of the node editor, click Run.

Next steps

Configure node scheduling: If you need to run a node periodically, configure its Scheduling Policy in the Scheduling Settings panel on the right.
Publish a node: To run a task in the production environment, click the icon to publish the node. A node runs on schedule only after it is published to the production environment.
Task O&M: After a task is published, you can monitor the status of its periodic runs in the Operation Center. For more information, see Get started with Operation Center.