Create an EMR Spark SQL node-DataWorks(DataWorks)-阿里云帮助中心

Prerequisites

Before developing a node, if you need to customize the component environment, you can create a custom image based on the official dataworks_emr_base_task_pod image and use the image in DataStudio.

For example, you can replace Spark JAR packages or include specific libraries, files, or JAR packages when you create a custom image.
Register an EMR cluster with DataWorks. For more information, see Legacy Data Development: Bind EMR compute resources.
RAM users must have the required permissions. A workspace owner or administrator must add the RAM user to the workspace and assign the Development or Workspace Administrator role. The Workspace Administrator role grants extensive permissions. Assign this role with caution. For more information, see Add members to a workspace.
Purchase and configure a resource group, including binding it to a workspace and configuring its network settings. For more information, see Use a serverless resource group.
Create a business flow. In DataStudio, all development nodes are organized into business flows. For more information, see Create a business flow.
If your tasks require a specific development environment, use DataWorks to build a custom image with the necessary components. For more information, see Custom images.

Limitations

This type of task can run only on a serverless resource group (recommended) or an exclusive resource group for scheduling. If you need to use a custom image in DataStudio, you must use a serverless resource group.
To manage metadata for a DataLake cluster or custom cluster in DataWorks, configure EMR-HOOK on the cluster. Without this configuration, DataWorks cannot display real-time metadata, generate audit logs, provide data lineage, or perform EMR-related governance tasks. For more information about how to configure EMR-HOOK, see Configure EMR-HOOK for Spark SQL.
Data lineage is supported for EMR Serverless Spark clusters, but not for EMR on ACK Spark clusters.
Visual function registration is supported for DataLake clusters and custom clusters but not for EMR on ACK Spark clusters or EMR Serverless Spark clusters.

Notes

If Ranger access control is enabled for Spark in the EMR cluster bound to your workspace:

This feature is supported by default for Spark tasks that use the default image.
To run Spark tasks using a custom image, submit a ticket to have the image upgraded to support this feature.

Step 1: Create a node

Go to the DataStudio page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Development and O&M > Data Development. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.
Create an EMR Spark SQL node.
1. Right-click the target business flow and choose Create Node > EMR > EMR Spark SQL.
  
  Note
  You can also hover over Create and choose Create Node > EMR > EMR Spark SQL.
2. In the Create Node dialog box, set the Name, Compute Engine Instance, Node Type, and Path. Click Confirm to open the node editor.
  
  Note
  The node name can contain uppercase letters, lowercase letters, Chinese characters, digits, underscores (_), and periods (.).

Step 2: Develop the task

In the EMR Spark SQL node editor, double-click the created node to open the task development page.

Write SQL code

You can define variables using the ${variable_name} format and assign their values in the Scheduling Settings > Scheduling Parameter pane. This enables dynamic parameter passing during scheduled runs. For more information about supported formats, see Formats of scheduling parameters. The following sample code is provided.

SHOW TABLES; 
-- Use ${var} to define a variable named var. If you assign the ${yyyymmdd} value to this variable, you can create a table whose name is suffixed with the business date.
CREATE TABLE IF NOT EXISTS userinfo_new_${var} (
ip STRING COMMENT'IP address',
uid STRING COMMENT'User ID'
)PARTITIONED BY(
dt STRING
); -- You can use this variable together with scheduling parameters.

Note

The size of an SQL statement cannot exceed 130 KB.
If multiple EMR compute resources are bound to your workspace, select the one required for your task. This step is unnecessary if only one resource is bound.

(Optional) Advanced settings

You can configure Spark-specific properties in the advanced settings for the node. For more information about Spark properties, see Spark Configuration. The available advanced parameters vary by EMR cluster type, as described in the following tables.

Datalake and custom: EMR on ECS

Parameter	Description
queue	The scheduling queue to which jobs are submitted. Default value: default. For more information about EMR YARN, see Basic queue configurations.
priority	The priority. Default value: 1.
FLOW_SKIP_SQL_ANALYZE	The SQL execution mode. Valid values: `true`: Runs multiple SQL statements per execution. `false` (default): Runs one SQL statement per execution. Note This parameter can be used only for test runs in the development environment.
ENABLE_SPARKSQL_JDBC	The method used to submit SQL code. Valid values: `true`: Submits SQL code by using the JDBC method. If an EMR cluster does not have the Kyuubi service, the code is submitted to Spark Thrift Server. If an EMR cluster has the Kyuubi service, the code is submitted to Kyuubi, and you can specify custom Spark parameters. Both submission methods support data lineage. However, for tasks submitted to the Thrift Server, metadata output is not available. `false` (default): Indicates that the SQL code is submitted in Spark-submit cluster mode. In this submission mode, both Spark 2 and Spark 3 support metadata lineage and output information. You can also specify custom Spark parameters. Note When you use the spark-submit cluster mode, temporary files and directories are created in the `/tmp` directory of the cluster's HDFS by default. Ensure that you have read and write permissions on this directory. If you use the spark-submit cluster mode, you can directly append custom SparkConf parameters in the advanced settings. When you submit the code, DataWorks automatically adds the new parameters to the command. Example: `"spark.driver.memory" : "2g"`.
DATAWORKS_SESSION_DISABLE	This parameter applies only to test runs in the development environment. Valid values: `true`: A new JDBC connection is created for each SQL statement run. `false` (default): The same JDBC connection is reused when you run different SQL statements in a node. Note When this parameter is set to `false`, the Hive `yarn applicationId` is not printed. To print the `yarn applicationId`, set this parameter to `true`.
Other	Add custom Spark Configuration parameters. Use the following format: `"spark.eventLog.enabled":false` . DataWorks automatically appends these parameters to the submission command using the `--conf key=value` format. For more information about parameter configurations, see Set global Spark parameters. Note DataWorks allows you to set global Spark parameters at the workspace level for different DataWorks modules. You can specify whether the global Spark parameters take precedence over the Spark parameters within a specific module. To enable Ranger access control, add the `spark.hadoop.fs.oss.authorization.method=ranger` configuration in Set global Spark parameters. This ensures that Ranger-based access control takes effect.

EMR serverless spark

For more information about parameter settings, see Set parameters for submitting a Spark job.

Parameter	Description
FLOW_SKIP_SQL_ANALYZE	The SQL execution mode. Valid values: `true`: Runs multiple SQL statements per execution. `false` (default): Runs one SQL statement per execution. Note This parameter can be used only for test runs in the development environment.
DATAWORKS_SESSION_DISABLE	Specifies the task submission method. When you run a task in Data Development, the task is submitted to SQL Compute by default. You can use this parameter to specify whether to submit the task to SQL Compute or to a queue. `true`: The task is submitted to a queue. By default, the queue that is specified when you bind the compute resource is used. If you set the `DATAWORKS_SESSION_DISABLE` parameter to `true`, you can configure the `SERVERLESS_QUEUE_NAME` parameter to specify the queue to which the task is submitted when it is run in Data Development. `false` (default): The task is submitted to SQL Compute for execution. Note This parameter takes effect only for runs in the Data Development environment, not for scheduled runs.
SERVERLESS_RELEASE_VERSION	The Spark engine version. By default, the Default Engine Version configured for the cluster on the Cluster Management page of the Management Center is used. Set this parameter to override the default for a specific task. Note The `SERVERLESS_RELEASE_VERSION` parameter in the advanced settings takes effect only when the SQL Compute session specified for the registered cluster is not running in the EMR Serverless Spark console.
SERVERLESS_QUEUE_NAME	The resource queue to which tasks are submitted. When a task is submitted to a queue, the Default Resource Queue configured for the cluster on the Cluster Management page of the Management Center is used by default. If you require resource isolation and management, you can add queues. For more information, see Manage resource queues. Configuration options: Specify the resource queue by setting node parameters. Specify the resource queue by setting global Spark parameters. Note The `SERVERLESS_QUEUE_NAME` parameter in the advanced settings takes effect only when the SQL Compute session specified for the registered cluster is not running in the EMR Serverless Spark console. When running in Data Development: You must first set the `DATAWORKS_SESSION_DISABLE` parameter to `true` to submit tasks to a queue. This allows the `SERVERLESS_QUEUE_NAME` parameter to take effect. When running on a schedule from Operation Center: Tasks are always submitted to a queue, not to SQL Compute.
SERVERLESS_SQL_COMPUTE	The SQL Compute session that you want to use. By default, the Default SQL Compute that is configured for the cluster on the Cluster Management page of the Management Center is used. If you want to use different SQL Compute sessions for different tasks, you can specify them here. For more information about how to create and manage SQL Compute sessions, see Manage SQL Compute sessions.
Other	Add custom Spark Configuration parameters. Use the following format: `"spark.eventLog.enabled":"false"`. DataWorks automatically adds the parameters to the code sent to the EMR cluster in the `--conf key=value` format. Note DataWorks allows you to set global Spark parameters at the workspace level for different DataWorks modules. You can specify whether the global Spark parameters take precedence over the Spark parameters within a specific module. For more information, see Set global Spark parameters.

Save and run the task

In the toolbar, click the Save icon to save your SQL statements, and click the Run icon to run the SQL task.

In the run dialog box, select a resource group that has a successful network connection to your Spark service. If the node code uses variables, assign constant values to the variables during the debug run. To configure the resource group and parameters for scheduled runs, see Configure node scheduling. For more information about task debugging, see Task debugging process.

Note

If you need to change the values of parameters in the code, click Advanced Run in the toolbar. For more information about the parameter value assignment logic, see What are the differences in value assignment logic among Run, Advanced Run, and smoke testing in the development environment?.

Spark: EMR on ACK

Parameter

Description

FLOW_SKIP_SQL_ANALYZE

The SQL execution mode. Valid values:

true: Runs multiple SQL statements per execution.
false (default): Runs one SQL statement per execution.

Note

This parameter can be used only for test runs in the development environment.

Other

Add custom Spark Configuration parameters.

Use the following format: "spark.eventLog.enabled":false. DataWorks automatically adds the parameters to the code sent to the EMR cluster in the --conf key=value format.

Note

DataWorks allows you to set global Spark parameters at the workspace level for different DataWorks modules. You can specify whether the global Spark parameters take precedence over the Spark parameters within a specific module. For more information, see Set global Spark parameters.

Hadoop: EMR on ECS

Parameter	Description
queue	The scheduling queue to which jobs are submitted. Default value: default. For more information about EMR YARN, see Basic queue configurations.
priority	The priority. Default value: 1.
FLOW_SKIP_SQL_ANALYZE	The SQL execution mode. Valid values: `true`: Runs multiple SQL statements per execution. `false` (default): Runs one SQL statement per execution. Note This parameter can be used only for test runs in the development environment.
USE_GATEWAY	Specifies whether to submit the node's job through a gateway cluster. Valid values: `true`: The job is submitted through a gateway cluster. `false` (default): The job is not submitted through a gateway cluster. By default, the job is submitted to a header node. Note If no gateway cluster is associated with the cluster where the node resides, subsequent EMR job submissions fail if you manually set this parameter to `true`.
Other	Add custom Spark Configuration parameters. Use the following format: `"spark.eventLog.enabled":false`. DataWorks automatically adds the parameters to the code sent to the EMR cluster in the `--conf key=value` format. For more information about parameter configurations, see Set global Spark parameters. Note DataWorks allows you to set global Spark parameters at the workspace level for different DataWorks modules. You can specify whether the global Spark parameters take precedence over the Spark parameters within a specific module. To enable Ranger access control, add the `spark.hadoop.fs.oss.authorization.method=ranger` configuration in Set global Spark parameters. This ensures that Ranger-based access control takes effect.

Run the task

In the toolbar, click the Run icon. In the Parameter dialog box, select the created scheduling resource group and click Running.
Note
- To access compute resources in a public network or a Virtual Private Cloud (VPC), you must use a scheduling resource group that has passed a connectivity test with the compute resources. For more information, see Network connectivity.
- If you need to change the resource group for subsequent task runs, you can click the Running with Parameters icon and select the desired resource group.
- When you use an EMR Spark SQL node to query data, a maximum of 10,000 records can be returned, and the total data size cannot exceed 10 MB.
Click the Save icon to save the SQL statements.
(Optional) Perform a smoke test.

If you want to perform a smoke test in the development environment, you can run the smoke test after you run or commit the node. For more information, see Perform a smoke test.

Step 3: Configure scheduling

If you want the system to periodically run a task on the node, you can click Properties in the right-side navigation pane on the configuration tab of the node to configure task scheduling properties based on your business requirements. For more information, see Overview.

Note

You must set the Rerun attribute and Parent Nodes properties before submitting the node.
If you need to customize the component environment, you can create a custom image based on the official dataworks_emr_base_task_pod image and use the image in DataStudio.

For example, you can replace Spark JAR packages or include specific libraries, files, or JAR packages when you create a custom image.

Step 4: Deploy the task

After a task on a node is configured, you must commit and deploy the task. After you commit and deploy the task, the system runs the task on a regular basis based on scheduling configurations.

Click the icon in the top toolbar to save the task.
Click the icon in the top toolbar to commit the task.

In the Submit dialog box, configure the Change description parameter. Then, determine whether to review task code after you commit the task based on your business requirements.
Note
- You must configure the Rerun and Parent Nodes parameters on the Properties tab before you commit the task.
- You can use the code review feature to ensure the code quality of tasks and prevent task execution errors caused by invalid task code. If you enable the code review feature, the task code that is committed can be deployed only after the task code passes the code review. For more information, see Code review.

If you use a workspace in standard mode, you must deploy the task in the production environment after you commit the task. To deploy a task on a node, click Deploy in the upper-right corner of the configuration tab of the node. For more information, see Deploying tasks.

More operations

After you commit and deploy the task, the task is periodically run based on the scheduling configurations. You can click Operation Center in the upper-right corner of the configuration tab of the corresponding node to go to Operation Center and view the scheduling status of the task. For more information, see Manage auto triggered tasks.

FAQ

Why does the error message "DlfMetaStoreClientFactory not found" appear when I use spark-submit in YARN-Cluster mode in an EMR Spark node after I enable Kerberos for the EMR cluster?
Before submitting an EMR Spark SQL task to SQL Compute from Data Development, verify that the SQL Compute session is in the Running state. Otherwise, the task will fail. For more information about how to check the status, see Manage SQL Compute sessions.