How to use CDP or CDH on DataWorks-DataWorks(DataWorks)-阿里云帮助中心

Background information

CDH is Cloudera's open-source platform distribution. It includes out-of-the-box features like cluster management, monitoring, and diagnostics, and supports various components for running end-to-end big data workflows.
CDP is a public data platform that collects and consolidates customer data from multiple platforms. It gathers real-time data to create individual user data profiles.

You can register CDH and CDP clusters in DataWorks for task development, scheduling, metadata management in Data Map, and Data Quality checks.

Limitations

You can run tasks on CDH or CDP clusters by using only serverless resource groups (recommended) or old-version exclusive resource groups for scheduling.
Note
- (Recommended) A serverless resource group is a general-purpose resource group that can be used for various task types, such as data synchronization and task scheduling. To learn how to purchase a serverless resource group, see Use serverless resource groups. If you have purchased an old-version exclusive resource group for scheduling, you can use it to run tasks on your CDH or CDP cluster.
- New users can purchase only serverless resource groups.
- If you register a cluster of a Custom Version to DataWorks, you can use only an old-version exclusive resource group for scheduling. For more information about cluster versions, see Step 2: Register a CDH or CDP cluster.
You can register a CDH or CDP cluster to DataWorks only in the following regions: China (Beijing), China (Shanghai), China (Hangzhou), China (Shenzhen), China (Zhangjiakou), China (Chengdu), and Germany (Frankfurt).

Prerequisites

DataWorks is activated. For more information, see Activate DataWorks.
A CDP or CDH cluster is deployed and registered.

DataWorks supports CDP or CDH clusters deployed in environments other than Alibaba Cloud ECS. However, the cluster's environment must be able to connect to an Alibaba Cloud VPC. You can use solutions such as Express Connect or VPN to ensure network connectivity. For more information, see Bind a CDH or CDP compute resource.
A serverless resource group is purchased.

By default, a purchased serverless resource group is not connected to the networks of other cloud services. To use a CDP or CDH cluster, you must first establish network connectivity between the cluster and the serverless resource group. For more information about how to purchase a resource group, see Use serverless resource groups.
A DataWorks workspace is created. For more information, see Configure a workspace.

Usage notes

The following table outlines the CDP/CDH development workflow in DataWorks.

Item	Description
Billing	When you develop CDP or CDH tasks on DataWorks, fees are incurred for DataWorks and other services.
Prerequisites	Before you develop CDP or CDH tasks on DataWorks, purchase a DataWorks edition and the required resource groups, and complete CDP or CDH cluster registration and development environment preparation.
Access control	DataWorks provides product-level and module-level access control, allowing you to grant different permissions to different users based on your business requirements.
Get started: Data Integration	DataWorks Data Integration allows you to read data from and write data to CDP/CDH Hive, and supports multiple data synchronization scenarios, such as batch synchronization and full and incremental synchronization tasks.
Get started: Data development and operations	DataWorks provides a data modeling service that organizes unstructured, scattered, and complex data into a well-structured, manageable form. It also offers Data Studio for developing scheduled tasks, which works with Operation Center for task monitoring and O&M.
Get started: Data Governance	DataWorks provides CDP and CDH metadata management and Data Governance capabilities.
Get started: Data Analysis and Data Service	DataWorks Data Analysis provides data analysis and service sharing capabilities for CDP and CDH.
Get started: Open Platform	DataWorks provides open APIs that help you integrate applications with DataWorks for data workflow management, data governance, and O&M, enabling timely responses to business status changes.

Billing

1. DataWorks charges

These charges appear on your DataWorks bill. For more information about DataWorks billing, see Billable items of DataWorks.

Charge	Description
DataWorks edition fee	Before you can develop tasks, you must activate DataWorks. Standard Edition, Professional Edition, and Enterprise Edition require a paid subscription.
Scheduling resource fee	Tasks require scheduling resources to run on a schedule. You can use a serverless resource group (recommended) or an old-version exclusive resource group. Note A serverless resource group can be used for both task scheduling and data synchronization.
Synchronization resource fee	Data synchronization tasks require dedicated synchronization resources in addition to scheduling resources. You can use a serverless resource group (recommended) or an old-version exclusive resource group for data integration.

Note

No scheduling fees are charged for tasks that are run using the Run or Run with Parameters feature on the DataStudio page.
No scheduling fees are charged for tasks that fail to run or are run as dry runs.

For information about how DataWorks dispatches scheduling tasks to help you better understand the billing details, see How DataWorks dispatches scheduling tasks.

II. Non-DataWorks fees

The following fees are not included in DataWorks bills.

Important

Fees incurred by other services are determined by the billing rules of the corresponding services. You can view the billing documentation of the corresponding services for details. For more information, see Billing.

Fee	Description
Database fees	When you synchronize data, fees may be incurred for reading data from and writing data to upstream and downstream databases.
Compute and storage fees	When you run compute engine tasks, compute and storage fees of the compute engine may be incurred.
Network service fees	When you connect DataWorks to other services over the network, network service fees may be incurred. For example, if you use Express Connect, shared bandwidth, or EIP to establish network connectivity, the corresponding service fees are incurred.

Prerequisites

I. Resource preparation

Category	Description	References
Edition selection	DataWorks Basic Edition can meet the basic requirements of CDP or CDH for data migration to the cloud, data development and scheduling, and simple Data Governance. To access more professional Data Governance and Data Security solutions, you can choose the Standard Edition, Professional Edition, or Enterprise Edition.	DataWorks: Features by edition
Resource group selection	CDP or CDH clusters support only serverless resource groups (recommended) or old-version exclusive resource groups for scheduling.	Use serverless resource groups

II. Development environment preparation

You must register a CDP or CDH cluster in a DataWorks workspace before you can perform data development in Data Studio, and manage workspace members for collaborative development.

Category	Description	References
Data synchronization environment preparation	Before you run data synchronization tasks based on the Hive component of a cluster, you must create a corresponding DataWorks data source for the component.	Supported data sources and synchronization solutions
Data Studio and Data Analysis environment preparation	Before you use DataWorks for periodic scheduling of compute engine tasks, you must add the cluster to DataWorks. After the cluster is added, you can use it for data development, Data Analysis, and periodic task scheduling.	DataStudio (legacy): Register a CDH computing resource
Collaborative development environment preparation	To enable RAM users to collaborate within a workspace, perform the following operations: Add the RAM users who need to collaborate to the workspace as members and assign them the developer role. Add the workspace members to the CDP or CDH cluster environment.	Add workspace members

Access control

DataWorks provides product-level and module-level access control. You can grant different permissions to different users based on your business requirements.

I. Data access control

For RAM users who are added to a DataWorks workspace to develop CDP or CDH tasks, you can configure cluster account mappings so that workspace members (RAM users) have the permissions of the mapped cluster accounts. For more information about cluster account mappings, see Configure cluster account mappings.

II. Module-level access control

Before you start data development, you can refer to Grant permissions to RAM users to grant different operation permissions to RAM users. The following types of permissions are available:

Use RAM policies to manage permissions on DataWorks modules (for example, deny a user access to Data Map) and the DataWorks console (for example, allow a user to delete a workspace).
Use workspace roles to manage permissions on workspace-level modules (for example, allow a user to access Data Studio for development) and global modules (for example, deny a user access to the Data Security Guard module).

Get started

DataWorks provides multiple modules. You can develop scheduling tasks in Data Studio and monitor them in Operation Center. DataWorks also provides process controls for task development and deployment to help you standardize operations and ensure security.

I. Data Integration

Data Integration allows you to read data from and write data to CDP/CDH Hive and CDP/CDH HBase. Create the Hive or HBase component as a DataWorks data source to synchronize data between the component and other data sources. You can use batch synchronization or full and incremental synchronization tasks. For more information, see Data Integration.

II. Data development and operations

Module	Description	References
Data Modeling	Data modeling is the first step in end-to-end data governance. Drawing on Alibaba's data platform modeling methodology, it covers data warehouse planning, data standards, dimensional modeling, and data metrics to interpret business data from a business perspective, enabling consistent data understanding across the organization.	Data Modeling
Data Studio	DataWorks integrates CDP and CDH compute engines, allowing you to run data synchronization and data development tasks. Data synchronization: Data Studio supports only some batch synchronization and real-time synchronization scenarios. For more data synchronization options, go to Data Integration. Data development: You can develop various tasks and configure periodic scheduling in DataWorks without using complex command lines.	Create a CDH Hive node Create a CDH Spark node Create a CDH MR node Create a CDH Presto node Create a CDH Impala node
	You can combine DataWorks general-purpose nodes and compute engine nodes for complex logic processing. The following are the main nodes: Virtual nodes to control business processes. HTTP trigger, OSS Object Check, and FTP Check nodes to trigger scheduling from external systems. Assignment and parameter nodes to pass parameters between upstream and downstream nodes. Do-while nodes to loop code, for-each nodes to iterate and evaluate the assignment results of upstream nodes, and branch nodes. Other nodes, such as general-purpose Shell nodes and MySQL database nodes.	Virtual nodes HTTP trigger nodes OSS Object Check nodes FTP Check nodes Assignment nodes Parameter nodes do-while node logic for-each node logic Branch nodes
	After you finish developing a node task, you can perform the following operations as needed: Scheduling configuration If the node task needs to run periodically, you can define its scheduling properties, such as scheduling dependencies and parameters. Node debugging To ensure that production tasks run efficiently and to prevent waste of computing resources, debug a task before you deploy it. Node deployment A task must be deployed to the production environment to be scheduled periodically. After deployment, you can go to Operation and Maintenance Center > Auto Triggered Node to view and manage the scheduled task. Node management You can deploy or undeploy node tasks, and batch-modify scheduling properties. Development process control DataWorks provides development process control for task development and deployment, such as code review, mandatory smoke testing, and custom check logic, to ensure the correctness and security of task-related operations.	Overview of task scheduling configuration Task debugging Deploy tasks Batch operations Development process control
Operation Center	Operation Center is a centralized O&M and monitoring platform that lets you view task running status in real time, perform intelligent diagnostics and reruns for abnormal tasks, and use smart baselines to ensure timely completion of critical tasks.	Basic O&M for auto-triggered nodes
Data Quality	Data Quality ensures data availability throughout the data development lifecycle. By validating quality rules integrated with the task scheduling workflow, it helps you identify and contain quality issues early, delivering reliable data for your business.	Data quality

III. Data Governance

After a CDP/CDH cluster is registered, DataWorks automatically collects metadata from your data sources. You can go to Data Map to view the metadata. You can also go to Data Asset Governance to view the governance issues detected by DataWorks and perform governance operations.

Module	Description	References
Data Map	Data Map provides an enterprise-level data management platform. Based on unified metadata, it offers data object management and inventory, lineage viewing, and the ability to quickly find and understand data objects. Note Table-level and column-level lineage is currently supported for CDH Hive, CDH Spark, CDH Spark SQL, and CDH Impala nodes. For more information, see Lineage.	Overview
Security Center Data Security Guard Approval center	Security Center, Data Security Guard, and the approval center provide an integrated data security governance interface that covers data asset classification, sensitive data identification, data access management, sensitive data masking, sensitive data access auditing, and risk identification and response. Note The approval center does not support custom approval workflows for CDH/CDP tables.	Security Center overview Overview Overview
Data Asset Governance	Data Asset Governance provides proactive, systematic governance across multiple domains by consolidating governance rules, automatically identifying optimization issues, and applying pre-event and post-event optimization policies. Note Only the global check items and governance items in Data Asset Governance can be used to govern CDH/CDP data issues. The specific supported features are subject to actual usage.	Overview

4. Data analysis and services

DataWorks provides data processing, analysis, and service capabilities, with centrally managed APIs for efficient data sharing.

Module	Description	Related documentation
DataAnalysis	Run online SQL queries, gain business insights, and edit and share data. You can also save query results as chart cards to build visual data reports.	DataAnalysis
DataService Studio	DataService Studio provides comprehensive data service and sharing capabilities, helping enterprises centrally manage API services for internal and external use.	DataService Studio

5. Open Platform

DataWorks Open Platform helps you integrate application systems with DataWorks for development process control, data governance, O&M, and real-time response to business status changes.

Category	Description	Related documentation
OpenAPI	OpenAPI provides APIs for interaction between on-premises services and DataWorks, improving big data processing efficiency while reducing manual operations, O&M overhead, data risks, and costs.	OpenAPI
OpenEvent	OpenEvent provides a message subscription service. By subscribing to DataWorks events, your applications receive real-time status changes for timely responses and personalized decision-making.	Overview of OpenEvent
Extensions	Extensions let you register a service program to intercept and respond to subscribed OpenEvent messages, enabling notifications and process control for specific events.	Overview of extensions