EMR Kyuubi node

Prerequisites

• Create an Alibaba Cloud EMR cluster and register it with DataWorks. For more information, see New Data Studio: Attach an EMR compute resource.

• (Optional, for RAM users) The Resource Access Management (RAM) user for task development must be added to the workspace and assigned the Development or Workspace Administrator role (this role includes extensive permissions and must be granted with caution). For more information, see Add workspace members.

  If you are using a root account, skip this step.

Limitations

These tasks can run only on a serverless resource group (recommended) or an exclusive scheduling resource group.

Procedure

1. On the EMR Kyuubi node editing page, perform the following operations.

   Develop SQL code

   In the SQL editor, develop the task code. You can define variables in your code in the ${variable_name} format. On the right-side panel, go to Scheduling Settings and assign values to these variables in the Scheduling Parameters field to pass dynamic parameters during scheduled runs. For more information, see Sources and expressions of scheduling parameters. Example:

   SHOW TABLES;
   SELECT * FROM kyuubi040702 WHERE age >= '${a}'; -- You can use scheduling parameters.

   Note

   The SQL statement cannot exceed 130 KB in size.

   (Optional) Configure advanced parameters

   On the right-side panel, go to Scheduling Settings > EMR Node Parameters > DataWorks parameters.

   Note

   To configure other open-source Spark properties, go to Scheduling Settings > EMR Node Parameters > Spark parameter on the right-side panel.

   Parameter	Description
   queue	The YARN resource queue for the job. Default: default. Note If a workspace-level YARN Resource Queue is configured when you register the EMR cluster with the DataWorks workspace, the queue selection rules for Kyuubi tasks are as follows: If Global Settings Take Precedence is set to Yes, the scheduling queue configured during EMR cluster registration is used. If Global Settings Take Precedence is not configured, the scheduling queue configured for the EMR Kyuubi node is used. For more information about EMR YARN, see Basic queue configurations. For more information about queue configuration during EMR cluster registration, see Set a global YARN resource queue.
   priority	The job priority. Default: 1.
   FLOW_SKIP_SQL_ANALYZE	The SQL statement execution mode. Valid values: true: Execute multiple SQL statements at a time. false (default): Execute one SQL statement at a time. Note This parameter applies only to test runs in the data development environment.
   DATAWORKS_SESSION_DISABLE	Controls JDBC connection behavior for direct test runs in the development environment. Valid values: true: Create a new JDBC connection for each SQL statement execution. false (default): Reuse the same JDBC connection for different SQL statements within the same node. Note If this parameter is set to false, the Hive yarn applicationId is not printed in the log. To print the yarn applicationId, set this parameter to true.

   Run the SQL task

   1. In the Compute Resource section of Run Configuration, select the Compute Resource and DataWorks Resource Group.

      Note

      • You can also adjust the CUs for Scheduling based on the resource requirements of the task. The default value is 0.25.

      • To access a data source over the public network or in a VPC, you must use a scheduling resource group with verified connectivity to that data source. For more information, see Network connectivity solutions.

   2. In the parameter dialog box in the toolbar, select the data source, and then click Run.

2. Configure the scheduling information for the node based on your business requirements. For more information, see Node scheduling configuration.

3. Deploy the node. For more information, see Deploy nodes and workflows.

4. After deployment, view the task run status in Operation Center. For more information, see Get started with Operation Center.

FAQ

• Q: The node fails with a connection timeout error. What can I do?

  A: Ensure that the resource group and the cluster can connect to each other over the network. Go to the computing resources list and click Resource Initialization. In the dialog box that appears, click Re-initialize. Verify that the initialization is successful.

  A message at the top of the dialog box indicates that you must initialize the resource group when you bind the cluster for the first time or when cluster service configurations change, such as modifications to hive-site.xml. Otherwise, tasks may fail. After initialization succeeds, the status column shows Successfully initialized with the completion time.