To use instances such as MaxCompute and Hologres for data development in DataWorks, you must first associate them as compute resources. You can then create and manage these resources to support task development and scheduling.
Relationship between compute resources and data sources
DataWorks supports various compute resources. After you associate a compute resource, you can develop complex data processing and periodic scheduling tasks in Data Studio. When you associate most compute resources with DataWorks, a data source with the same name is automatically created. You can then use this data source in the Data Integration module for data synchronization tasks. The differences between a compute engine and a data source are as follows:
-
A compute resource is a compute engine instance that executes data processing and analysis tasks.
-
A data source connects to data storage services to store and manage data.
When you perform ETL data transformations by using SQL nodes, such as converting a JSON column in an AnalyticDB for MySQL table into a new table, you must also associate the corresponding compute engine. SQL nodes in Data Studio require a compute engine to run SQL statements, even if you only use data transformation features.
Supported compute resources
DataWorks supports the following compute resources for data development.
|
Category |
Resource type |
Association guide |
Data Studio (new version) |
Data Studio (legacy version) |
|
Offline computing |
|
|
||
|
|
|
|||
|
Serverless Ray |
|
|
||
|
Real-time query |
|
|
||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
Real-time computing |
|
|
||
|
Multimodal search |
|
|
||
|
Cluster management |
|
|
||
|
|
|
When you associate a MaxCompute, AnalyticDB for MySQL, AnalyticDB for PostgreSQL, AnalyticDB for Spark, ClickHouse, Hologres, Lindorm, EMR Serverless StarRocks, or OpenSearch compute resource, a data source with the same name is automatically created in the current workspace.
Permission description
-
Only workspace members who have the operator or workspace administrator role, or members who have the AliyunDataWorksFullAccess or AdministratorAccess permission policy can create compute resources. For more information, see Manage permissions on workspace-level services and Grant permissions to a RAM user.
-
In addition to the preceding permissions, some compute resources may require additional permissions during creation. Follow the on-screen instructions to grant access.
Associate a compute resource
You can associate a compute resource from the corresponding entry based on whether the workspace has Use Data Studio (New Version) enabled.
Associate a compute resource by using Data Studio (new version)
-
Log on to the DataWorks console, switch to the target region, and then click in the left-side navigation pane. Select the target workspace from the drop-down list and click Go to Management Center.
-
In the left-side navigation pane, click Computing Resources to go to the compute resource list page. Refer to the corresponding documentation based on the type of compute resource you want to associate.
Associate a compute resource without Data Studio (new version)
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
-
In the left-side navigation pane, click the
icon to go to the Computing Resources list page. Refer to the corresponding documentation based on the type of compute resource you want to associate.-
Compute resource management: Click the Create Computing Resource button in the upper-right corner to create a compute resource.
-
Cluster management: Click the Computing Resources list page and then click the Create Cluster button in the upper-right corner to create a cluster engine.
Cluster management
Supported cluster version/type
Cluster association documentation
Associate a CDH/CDP cluster
DataWorks provides CDH5.16.2, CDH6.1.1, CDH6.2.1, CDH6.3.2, and CDP7.1.7 versions for direct selection. The component versions bundled with these cluster versions (that is, the version of each component in the cluster connection information) are fixed. If these cluster versions do not meet your business requirements, you can select Custom Version.
Associate an EMR cluster
Supported EMR cluster types: DataLake cluster (new data lake): EMR on ECS, Custom cluster: EMR on ECS, Hadoop cluster (legacy data lake): EMR on ECS, Spark cluster: EMR on ACK, and EMR Serverless Spark clusters.
Important-
The following EMR versions of Hadoop clusters (legacy data lake) are supported in DataWorks:
EMR-3.38.2, EMR-3.38.3, EMR-4.9.0, EMR-5.6.0, EMR-3.26.3, EMR-3.27.2, EMR-3.29.0, EMR-3.32.0, EMR-3.35.0, EMR-4.3.0, EMR-4.4.1, EMR-4.5.0, EMR-4.5.1, EMR-4.6.0, EMR-4.8.0, EMR-5.2.1, EMR-5.4.3
-
Hadoop clusters (legacy data lake) are no longer recommended. Migrate to DataLake clusters as soon as possible. For more information, see Migrate Hadoop clusters to DataLake clusters.
-
-
Disassociate a compute resource
Exercise caution when you disassociate a compute resource. The disassociation also deletes the data source with the same name. This may affect tasks that reference the compute resource or data source across modules such as Data Integration, Operation Center, Data Analysis, Data Service APIs, and Data Quality. Before you proceed, carefully read the on-screen prompts and migrate all tasks under the compute resource to another compute resource.
On the compute resource page, find the target compute resource and click Disassociate to disassociate it from the current workspace.
Appendix: Task execution environment description
In a standard mode workspace, a compute resource has two sets of configurations: one for the development environment and one for the production environment. Each environment can use a different database or instance. The system automatically routes access to the correct compute resource based on the runtime environment, isolating development from production. For example, a batch synchronization task in the development environment accesses the development database, while the same task in production accesses the production database.
-
A basic mode workspace has only one environment and cannot isolate development from production. For more information about workspace modes, see Workspace mode differences.
-
If you upgrade from basic mode to standard mode (workspaces that have Use Data Studio (new version) enabled do not support the upgrade), the original single compute resource is split into two compute resources isolated for the production environment and the development environment. For more information, see Upgrade a workspace mode.
