EMR Impala node

Prerequisites

• Create an Alibaba Cloud EMR cluster and register it with DataWorks. For more information, see New Data Studio: Attach an EMR compute resource.

• (Optional, for RAM users) The Resource Access Management (RAM) user for task development must be added to the workspace and assigned the Development or Workspace Administrator role (this role includes extensive permissions and must be granted with caution). For more information, see Add workspace members.

  If you are using a root account, skip this step.

• Configure a Hive data source in DataWorks and ensure that it passes the connectivity test. For more information, see Data Source Management.

Limitations

• You can run EMR Impala nodes only on a serverless resource group (recommended) or an exclusive resource group for scheduling.

• EMR Impala nodes can run only on the computing resources of a legacy data lake cluster (Hadoop). DataWorks no longer supports binding new Hadoop clusters. However, you can continue to use existing bound Hadoop clusters.

Procedure

1. On the EMR Impala node editor page, perform the following steps.

   Develop SQL code

   In the SQL editor, write the code for the task. You can define variables in your code by using the ${variable_name} format. Then, assign a value to the variable in the Scheduling Parameters section on the Scheduling Settings tab on the right side of the node editor page. This allows you to dynamically pass parameters for scheduled tasks. For more information on scheduling parameters, see Sources and expressions for scheduling parameters. The following code provides an example:

   SHOW  TABLES ;
   CREATE TABLE IF NOT EXISTS userinfo (
   ip STRING COMMENT'IP address',
   uid STRING COMMENT'User ID'
   )PARTITIONED BY(
   dt STRING
   ); 
   ALTER TABLE userinfo ADD IF NOT EXISTS PARTITION(dt='${bizdate}'); --This can be used with scheduling parameters.
   SELECT * FROM userinfo ;

   Note

   An SQL statement cannot exceed 130 KB in size.

   (Optional) Configure advanced parameters

   On the right side of the page, navigate to the Scheduling Settings tab. In the EMR Node Parameters > DataWorks parameters section, configure the specific parameters described in the following table.

   Note

   • The available advanced parameters vary by EMR cluster type.

   • To configure additional open-source Spark properties, use the EMR Node Parameters > Spark parameter section on the Scheduling Settings tab on the right side of the node.

   DataLake cluster/custom cluster: EMR on ECS

   Advanced parameter	Description
   FLOW_SKIP_SQL_ANALYZE	The execution mode of SQL statements. Valid values: true: Multiple SQL statements are executed at a time. false (default): Only one SQL statement is executed at a time. Note This parameter is applicable only to test runs in the development environment.
   DATAWORKS_SESSION_DISABLE	This parameter is applicable to test runs directly in the development environment. Valid values: true: A new JDBC connection is created each time an SQL statement is run. false (default): The same JDBC connection is reused when different SQL statements are run within the same node. Note When this parameter is set to false, the Hive yarn applicationId is not printed. To print the yarn applicationId, set this parameter to true.
   priority	The priority. Default value: 1.
   queue	The scheduling queue to which the job is submitted. Default value: default. For more information about EMR YARN, see Basic queue configurations.

   Hadoop cluster: EMR on ECS

   Advanced parameter	Description
   FLOW_SKIP_SQL_ANALYZE	The execution mode of SQL statements. Valid values: true: Multiple SQL statements are executed at a time. false (default): Only one SQL statement is executed at a time. Note This parameter is applicable only to test runs in the development environment.
   USE_GATEWAY	Specifies whether to submit the job through a Gateway cluster when this node submits a job. Valid values: true: The job is submitted through a Gateway cluster. false (default): The job is not submitted through a Gateway cluster. By default, the job is submitted to the header node. Note If the cluster to which this node belongs is not associated with a Gateway cluster and you manually set this parameter to true, subsequent EMR job submissions will fail.

   Run the SQL task

   1. In the Run Configuration section, select and configure a Compute Resource, configure the Compute Resource and Resource Group.

      Note

      • You can also configure CUs for Scheduling based on the resources required to run the task. The default value is 0.25 CU.

      • To access data sources in a public network or VPC network environment, use a resource group for scheduling that has passed the connectivity test with the data source. For more information, see Network connectivity.

   2. In the parameter dialog on the toolbar, select the Hive data source that you have created, and click Run to run the SQL task.

      Note

      When you use an EMR Impala node to query data, a maximum of 10,000 rows can be returned, and the total data size cannot exceed 10 MB.

   3. Click Save to save the node task.

2. To run the node task on a regular basis, configure the schedule settings based on your business requirements. For more information, see Configure schedule settings.

3. After the node task is configured, you must deploy the node. For more information, see Deploy a node.

4. After the task is deployed, you can view the running status of the scheduled task in Operation Center. For more information, see View scheduled tasks.

(Optional) View lineage information

To display table-level and column-level lineage for EMR Impala tasks in Data Map, you must first enable Impala lineage logs on the EMR cluster side. This feature is supported only for EMR DataLake clusters (both HMS and DLF metadata are supported). For more information, see Configure lineage for EMR Impala.

FAQ

• Q: The error "Impala JDBC Url is Empty" occurs?

  >>> [ERROR][LauncherFactory]: JobLauncher init Failed!
  java.lang.RuntimeException: Impala JDBC Url is Empty!
      at com.aliyun.emr.dataworks.dcc.launcher.type.ImpalaJobLauncher.initJdbcConnection

  A: Make sure that the Impala service has been added to the cluster. The Impala service is available only to existing users.

• Q: A connection timeout error occurs when the node runs?

  EMR execute task failed!
  FAILED: Build connection error! Could not open client transport with JDBC Uri: jdbc:hive2://<host>:21050/;auth=noSasl: java.net.ConnectException: Connection timed out (Connection timed out)

  A: Ensure that the resource group and the cluster can connect to each other over the network. Go to the computing resources list and click Resource Initialization. In the dialog box that appears, click Re-initialize. Verify that the initialization is successful.