How to use E-MapReduce in DataWorks-DataWorks(DataWorks)-阿里云帮助中心

DataWorks lets you create Hive, MR, Presto, Spark SQL, and other nodes based on the E-MapReduce (EMR) compute engine to configure EMR workflows, schedule tasks, and manage metadata. These capabilities help ensure efficient and stable data production and management. This topic describes the basic workflow for using EMR in DataWorks, including billing, environment preparation, and access control.

Background information

E-MapReduce (EMR) is an open source big data platform that provides a system solution for big data processing on Alibaba Cloud.

Built on open source Apache Hadoop and Apache Spark, EMR lets you use other systems in the Hadoop and Spark ecosystems to analyze and process data. Alibaba Cloud EMR offers EMR on ECS, EMR on ACK, and serverless deployment options to meet different requirements. For more information, see E-MapReduce overview.

Supported cluster types

Before you can run EMR tasks, you must create and register an EMR cluster with DataWorks. The following cluster types are supported: DataLake clusters (new data lake): EMR on ECS, Custom clusters: EMR on ECS, Hadoop clusters (legacy data lake): EMR on ECS, Spark clusters: EMR on ACK, and EMR Serverless Spark clusters.

Important

The following EMR versions of Hadoop clusters (legacy data lake) are supported in DataWorks:

EMR-3.38.2, EMR-3.38.3, EMR-4.9.0, EMR-5.6.0, EMR-3.26.3, EMR-3.27.2, EMR-3.29.0, EMR-3.32.0, EMR-3.35.0, EMR-4.3.0, EMR-4.4.1, EMR-4.5.0, EMR-4.5.1, EMR-4.6.0, EMR-4.8.0, EMR-5.2.1, EMR-5.4.3
Hadoop clusters (legacy data lake) are no longer recommended. We recommend that you migrate your Hadoop clusters to DataLake clusters as soon as possible. For more information, see Migrate a Hadoop cluster to a DataLake cluster.

Limitations

Permissions: Only RAM users or RAM roles with the following identities can register an EMR cluster. To grant these permissions, see Grant permissions to a RAM user.
- An Alibaba Cloud account.
- A RAM user or RAM role with the DataWorks Workspace Administrator role and the AliyunEMRFullAccess policy.
- A RAM user or RAM role with the AliyunDataWorksFullAccess and AliyunEMRFullAccess policies.
Region: EMR Serverless Spark is available only in China (Hangzhou), China (Shanghai), China (Beijing), China (Zhangjiakou), China (Ulanqab), China (Shenzhen), China (Chengdu), China (Hong Kong), Japan (Tokyo), Singapore, Indonesia (Jakarta), Germany (Frankfurt), and US (Virginia).
Task type: DataWorks does not support EMR Flink tasks.
Task execution: DataWorks supports running EMR tasks by using serverless resource groups (recommended) or legacy exclusive resource groups for scheduling.
Data governance:
- Only SQL tasks on EMR Hive, EMR Spark, and EMR Spark SQL nodes support data lineage generation. For clusters of version 5.9.1 or 3.43.1 or later, these nodes support both table-level and column-level data lineage.
  
  Note
  For Spark nodes, table-level and column-level data lineage are supported on EMR clusters of version 5.8.0 or 3.42.0 or later. On earlier versions, only Spark 2.x supports table-level data lineage.
- To manage metadata for a DataLake or custom cluster in DataWorks, you must first configure EMR-HOOK on the cluster. If EMR-HOOK is not configured, DataWorks cannot display metadata in real time, generate audit logs, or show data lineage. Currently, EMR-HOOK is supported only for the EMR Hive and EMR Spark SQL services. For more information, see Configure EMR-HOOK for Hive and Configure EMR-HOOK for Spark SQL.
For EMR clusters with Kerberos authentication enabled, you must add an inbound rule to the security group to allow UDP access from the CIDR block of the vSwitch that is associated with the resource group.

Note
Click the icon for Cluster Security Group on the Basic information tab of the EMR cluster to go to the Security Group Details page. On the Access Rules tab, click Inbound, and then click Added Manually. Set Protocol Type to Custom UDP. For Port Range, enter the KDC port specified in the /etc/krb5.conf file of the EMR cluster. Set Authorized object to the CIDR block of the vSwitch that is associated with the resource group.

Prerequisites

DataWorks is activated and a workspace is created. For more information, see Purchase guide and Configure a workspace.
An EMR cluster is created. For more information, see Create a cluster.

Note
You can use various EMR components to run tasks in DataWorks. The optimal configuration varies by component. When you create an EMR cluster, refer to EMR cluster configuration recommendations and select components based on your requirements.
A DataWorks serverless resource group is purchased.

By default, a newly purchased DataWorks resource group cannot connect to other Alibaba Cloud services. Before you use EMR, establish network connectivity between the EMR cluster and the resource group.
Note
- Serverless resource groups (recommended) are general-purpose resource groups that support various task types, such as data synchronization and task scheduling. For purchase instructions, see Use serverless resource groups. New users can purchase only serverless resource groups.
- If you have purchased a legacy exclusive resource group, you can also use it to run EMR tasks. Select the appropriate legacy exclusive resource group for the task type. For example, use an exclusive resource group for Data Integration to run data synchronization tasks and an exclusive resource group for scheduling to run scheduling tasks. For more information, see Use legacy resource groups.

Usage notes

The following table describes the main aspects of developing EMR tasks in DataWorks.

Item	Description
Billing	Developing EMR tasks in DataWorks incurs charges for DataWorks and other services.
Environment preparation	Before you develop EMR tasks in DataWorks, purchase the appropriate DataWorks edition and resource groups based on your business requirements, register your EMR cluster, and prepare the development environment.
Access control	DataWorks provides product-level and module-level access control. You can grant different permissions to users based on your business requirements for fine-grained access management.
Getting started: Data Integration	DataWorks Data Integration can read data from and write data to EMR Hive. It supports batch synchronization and full and incremental synchronization tasks.
Getting started: Data development and O&M	DataWorks provides Data Modeling to organize unstructured, disordered, complex, and large volumes of hard-to-manage data into a structured system. It also provides Data Studio for developing scheduling tasks. You can use Data Studio with Operation Center to monitor and maintain scheduling tasks.
Getting started: Data governance	DataWorks provides EMR metadata management and data governance capabilities.
Getting started: Data analysis and services	DataWorks Data Analysis provides EMR data analysis and service sharing capabilities.
Getting started: Open Platform	DataWorks provides open capabilities to help you integrate application systems with DataWorks, manage data processes, govern data, perform O&M, and promptly respond to changes in the business status of integrated application systems.

Billing

1. DataWorks charges

The following charges appear on your DataWorks bills. For more information about DataWorks billing, see Overview.

Billable item	Description
DataWorks edition fees	Activate DataWorks before you develop tasks. If you activate Standard Edition, Professional Edition, or Enterprise Edition, you must pay the corresponding edition fees.
Scheduling resource fees	After you develop tasks, scheduling resources are required to schedule them. You can use serverless resource groups (recommended) or legacy exclusive resource groups for scheduling and pay the corresponding resource group fees. Note A serverless resource group can be used for both task scheduling and data synchronization.
Data synchronization resource fees	Running data synchronization tasks requires data synchronization resources in addition to scheduling resources. You can use serverless resource groups (recommended) or legacy exclusive resource groups for Data Integration and pay the corresponding resource group fees.

2. Charges for other services

The following charges do not appear on your DataWorks bills.

Important

Charges for other services are determined by each service's billing rules. Refer to the billing documentation for the relevant service. For example, for EMR billing details, see Billing.

Billable item	Description
Database fees	Reading data from or writing data to upstream and downstream databases during data synchronization may incur database fees.
Computing and storage fees	Running tasks on a compute engine may incur computing and storage fees for that engine.
Network service fees	Establishing network connectivity between DataWorks and other services may incur network service fees. For example, using Express Connect, Internet Shared Bandwidth, or Elastic IP Address (EIP) to establish connectivity incurs fees for the corresponding services.

Environment preparation

1. Prepare resources

Category	Description	References
DataWorks edition	DataWorks Basic Edition supports basic EMR data migration to the cloud, data development and scheduled production, and simple data governance. For more advanced data governance and data security solutions, select Standard Edition, Professional Edition, or Enterprise Edition.	DataWorks: Features by edition
Resource group	EMR clusters support only serverless resource groups (recommended) or legacy exclusive resource groups for running tasks.	Use serverless resource groups Use legacy resource groups

2. Prepare the development environment

Register an EMR cluster with your DataWorks workspace before you develop data in Data Studio. Manage members at the workspace level to enable collaborative development.

Category	Description	References
Data synchronization environment	Before you run data synchronization tasks on a cluster component, add the component as the corresponding DataWorks data source.	Supported data sources and synchronization solutions
Data development and data analysis environment	Before you use DataWorks to periodically schedule compute engine tasks, add the cluster to DataWorks. You can then use the cluster for data development, data analysis, and periodic task scheduling.	Data Studio (legacy): Register an EMR cluster
Collaborative development environment	To enable RAM users to collaborate in a workspace, perform the following steps: Add the RAM users who need to collaborate as members of the workspace and assign them the Development role. Add the workspace members to the EMR cluster.	Manage permissions on workspace-level services Manage EMR users

Access control

DataWorks provides product-level and module-level access control. You can grant users different permissions based on your business requirements. The following sections describe these access control capabilities.

1. Data access control

For RAM users who join a DataWorks workspace to develop EMR tasks, configure cluster account mappings so that each workspace member has the permissions of the mapped cluster account. For more information about cluster account mappings, see Configure cluster identity mapping.

DataWorks provides visual interfaces for DLF permission requests, approvals, and audits to centrally manage permissions for a fully managed data lake. If your EMR cluster uses DLF as its metadata service, you can request and manage data access permissions in DataWorks Security Center. For more information, see DLF data access control.

2. Feature and module access control

Before you develop data, refer to Grant permissions to RAM users to grant the appropriate operation permissions. The following types of permissions are available:

Use Permission control for global services to manage access to DataWorks modules (for example, to prevent users from accessing Data Map) and DataWorks console operations (for example, to allow users to delete workspaces).
Use Manage permissions on workspace-level services to manage access to workspace-level modules (for example, to allow users to perform development operations in Data Studio) and global modules (for example, to prevent users from accessing Data Security Guard).

Getting started

DataWorks provides multiple modules. Develop scheduling tasks in Data Studio, then monitor and maintain them in Operation Center in the production environment. DataWorks also provides process controls for task development and deployment to standardize development operations and help ensure a secure development process.

1. Data Integration

DataWorks Data Integration lets you read data from and write data to EMR Hive. Add the Hive component as a DataWorks Hive data source to synchronize data from other data sources to Hive or from Hive to other data sources. You can choose batch synchronization or full and incremental synchronization tasks based on your requirements. For more information, see Data Integration.

2. Data development and O&M

Module	Description	References
Data Modeling	Data Modeling is the first step in end-to-end data governance. Based on Alibaba's data middle office modeling methodology, it interprets business data from four perspectives: data warehouse planning, data standards, dimensional modeling, and data metrics. This helps teams across an enterprise understand and share data consistently.	Data Modeling
Data Studio	DataWorks encapsulates EMR compute engine capabilities so that you can run EMR data synchronization and data development tasks. Data synchronization: Data Studio supports only some batch and real-time synchronization scenarios. For more synchronization capabilities, visit the Data Integration homepage. Data development: Develop and periodically schedule various tasks in DataWorks without using complex command-line tools.	Create an EMR Hive node Create an EMR MR node Create an EMR Spark SQL node Create an EMR Spark node Create an EMR Shell node Create an EMR Presto node Create an EMR Impala node Create an EMR Spark Streaming node Create an EMR Kyuubi node Create an EMR Trino node Create an EMR table Create and use EMR resources Create an EMR function
	Combine general-purpose DataWorks nodes with compute engine nodes to implement complex processing logic. The main node types are as follows: Virtual nodes to control business processes. HTTP trigger, OSS Object Check, and FTP Check nodes to trigger scheduling from external systems. Assignment and parameter nodes to pass parameters between upstream and downstream nodes. Do-while nodes to loop code, for-each nodes to iterate and evaluate the assignment results of upstream nodes, and branch nodes. Other nodes, such as general-purpose Shell nodes and MySQL database nodes.	Virtual nodes HTTP trigger nodes OSS Object Check nodes FTP Check nodes Assignment nodes Parameter nodes do-while node logic for-each node logic Branch nodes
	After you develop node tasks, you can perform the following operations as needed: Scheduling configuration If the node task needs to run periodically, you can define its scheduling properties, such as scheduling dependencies and parameters. Node debugging To ensure that production tasks run efficiently and to prevent waste of computing resources, debug a task before you deploy it. Node deployment A task must be deployed to the production environment to be scheduled periodically. After deployment, you can go to Operation and Maintenance Center > Auto Triggered Node to view and manage the scheduled task. Node management You can deploy or undeploy node tasks, and batch-modify scheduling properties. Development process control DataWorks provides development process control for task development and deployment, such as code review, mandatory smoke testing, and custom check logic, to ensure the correctness and security of task-related operations.	Overview of task scheduling configuration Task debugging Deploy tasks Batch operations Development process control
Operation Center	Operation Center is a one-stop big data O&M and monitoring platform. You can view task status in real time and perform O&M operations, such as intelligent diagnostics and task reruns, on abnormal tasks. Its intelligent baseline feature helps ensure timely task output by addressing unpredictable completion times for important tasks and the challenges of monitoring large numbers of tasks.	Basic O&M for auto-triggered nodes
Data Quality	Data Quality helps ensure data availability throughout the data development lifecycle. By efficiently validating data quality rules and integrating closely with task scheduling, it helps you detect quality issues promptly and prevent them from affecting downstream data, providing efficient, reliable, and trustworthy data for your business.	Overview

3. Data governance

After you register an EMR cluster with DataWorks, DataWorks automatically collects metadata from the engine. You can view the metadata in Overview. You can also go to Overview to view the governance issues detected by DataWorks and perform data governance operations.

Module	Description	References
Data Map	DataWorks Data Map is an enterprise-grade data management platform built on unified metadata. It lets you manage and inventory data objects, quickly find them, and gain a deeper understanding of them.	Overview
Security Center Data Security Guard Approval Center	Security Center provides a one-stop interface for data security governance. It covers data asset classification and grading, sensitive data detection, data authorization management, sensitive data masking, sensitive data access auditing, and risk detection and response.	Security Center overview Overview Overview
Data Governance Center	Data Governance Center helps you proactively and systematically govern data across multiple domains. It uses accumulated governance rules to automatically identify data assets that need optimization and provides both preventive and corrective governance strategies.	Overview

4. Data analysis and services

DataWorks data analysis and services provide data processing and analysis capabilities and let you efficiently share and access data through centrally managed APIs.

Module	Description	Related documentation
DataAnalysis	Run online SQL queries, gain business insights, and edit and share data. You can also save query results as chart cards to build visual data reports.	DataAnalysis
DataService Studio	DataService Studio provides comprehensive data service and sharing capabilities, helping enterprises centrally manage API services for internal and external use.	DataService Studio

5. Open Platform

DataWorks provides open capabilities to help you integrate application systems with DataWorks, manage data processes, govern data, perform O&M, and promptly respond to business status changes in the integrated systems.

Category	Description	References
OpenAPI	The OpenAPI feature of DataWorks Open Platform provides APIs that let local services interact with DataWorks. These APIs improve enterprise big data processing efficiency, reduce manual operations and O&M work, and lower data risks and business costs.	OpenAPI
OpenEvent	The OpenEvent feature of DataWorks Open Platform provides a message subscription service. Subscribe to DataWorks events and integrate your application systems with DataWorks to receive content status changes in real time, respond to events promptly, and support your own decision-making requirements.	Overview
Extensions	DataWorks provides message subscriptions through OpenEvent. Register your service programs as DataWorks extensions to intercept and respond to subscribed events, enabling notifications and process controls for specific events.	Overview

Appendix: EMR cluster configuration recommendations

You can use various EMR components to run tasks in DataWorks. The optimal configuration varies by component. When you configure an EMR cluster, refer to the following recommendations and select appropriate settings based on your requirements.

Kyuubi

When you configure Kyuubi in a production environment, set the memory for kyuubi_java_opts to 10g or more, and the memory for kyuubi_beeline_opts to 2g or more.
Spark
- Spark has relatively low default memory settings. Add memory configuration options to the spark-submit command to increase the default memory to an appropriate size.
- Adjust the following Spark settings based on the size of your EMR cluster: spark.driver.memory, spark.driver.memoryOverhead, spark.executor.memory.
Important
Only EMR Hive, EMR Spark, and EMR Spark SQL nodes in DataWorks support data lineage. EMR Hive nodes support both table-level and column-level lineage. Spark-type nodes support only table-level lineage.

For more information about Spark configuration, see Spark Memory Management.
HDFS

Adjust the following HDFS settings based on the size of your EMR cluster: hadoop_namenode_heapsize, hadoop_datanode_heapsize, hadoop_secondary_namenode_heapsize, hadoop_namenode_opts.