Create a single-tenant, multi-engine compute source

Prerequisites

• To create an offline compute source, you must first create a corresponding cluster. In this mode, you can create an offline compute source only by referencing a cluster. For more information about how to create a cluster, see Create a cluster in single-tenant multi-engine mode.

• To create a real-time compute source, you must first select a corresponding real-time compute engine.

Procedure

1. In the top navigation bar of the Dataphin homepage, choose Plan > Compute Source.

2. On the Compute Source page, click Add Compute Source. On the Create Compute Source page, configure the following parameters.

   In single-tenant and multi-engine mode, you can configure a compute source only by selecting Reference Specified Cluster. The available configuration items vary based on your selections.

   • Basic information of the compute source

     Parameter	Description
     compute source type	For an offline compute source, you can only select a type that has a corresponding existing cluster. The supported types include: MaxCompute AnalyticDB for PostgreSQL Alibaba Cloud EMR 3.x Alibaba Cloud EMR 5.x CDH 5.x CDH 6.x Cloudera Data Platform 7.x Huawei FusionInsight 8.x AsiaInfo DP 5.3 StarRocks Databricks Amazon EMR SelectDB Doris GaussDB (DWS) Transwarp TDH 6.x Transwarp TDH 9.3.x Transwarp ArgoDB Lindorm (compute engine) Hologres OushuDB Alibaba Cloud EMR Serverless Spark Flink Ververica Flink FusionInsight Flink
     compute source name	The name must meet the following requirements: It can contain Chinese characters, letters, digits, underscores (_), hyphens (-), periods (.), at signs (@), tildes (~), single quotation marks ('), parentheses (()), and spaces. It must be no more than 64 characters long.
     cluster	For an offline compute source, select an existing cluster that corresponds to the selected type.
     data lake table format	This option is disabled by default. If enabled, you can select a data lake table format. If the compute engine is Cloudera Data Platform 7.x, you can select Hudi. If the compute engine is Alibaba Cloud EMR 5.x, you can select Iceberg or Paimon. Note This parameter is available only when the compute source type is Cloudera Data Platform 7.x or Alibaba Cloud EMR 5.x.
     compute source description	Enter a brief description of the compute source, up to 128 characters.

   • Other configurations of the compute source

     Hadoop

     Hadoop compute sources include CDH 5.x, CDH 6.x, Cloudera Data Platform 7.x, Alibaba Cloud EMR 3.x, Alibaba Cloud EMR 5.x, AsiaInfo DP 5.3, and Huawei FusionInsight 8.x.

     In this mode, configuring a Hadoop compute source—including its queues, Hive and Spark services, and Impala tasks—is the same as when referencing a cluster in single-engine mode. For more information, see Hadoop compute source.

     MaxCompute

     Parameter	Description
     MaxCompute project	This option is available when the authentication method is RAM role-based proxy. Select or enter a MaxCompute project. The drop-down list displays only the projects visible to the specified AccessKey user. You can manually enter another project name or go to the MaxCompute console to create one. Note This parameter refers to the name of a MaxCompute project, not the name of a DataWorks workspace. In the MaxCompute console, switch the region in the upper-left corner. Then, you can view the MaxCompute Project Name on the Project Management tab.
     external project	Select whether this is an external project. By default, it is not an external project. If you select Yes, it is considered an external project. Note Integration tasks do not support reading data from or writing data to external MaxCompute projects. Hologres engines cannot be used to accelerate queries on external MaxCompute projects. The storage size of tables in an external project cannot be obtained from MaxCompute metadata. Storage health scores and storage size are not supported for these resources.
     compute project	Select or enter a compute project. The drop-down list displays only the projects visible to the specified AccessKey user. You can manually enter another project name or go to the MaxCompute console to create one. If the selected MaxCompute project is external, you must specify a different project to use for compute resources.

     AnalyticDB for PostgreSQL and OushuDB

     Parameter	Description
     JDBC URL	The JDBC URL configured in the selected cluster. This field is read-only.
     database	Select or enter a database.
     schema	Select or enter the database schema.
     priority task queue	You can select Use default execution user or Custom. Use default execution user: Uses the default execution user configured in the selected cluster. Custom: Includes five priority levels: Highest, High, Medium, Low, and Lowest. For each priority level, you can select Use cluster's default execution user or Custom. If you select Custom, you must enter an execution username and password. Note In AnalyticDB for PostgreSQL, resource queues are used for priority management. To create a custom priority task queue, you must first create a username and password for each priority level in AnalyticDB for PostgreSQL.

     StarRocks, SelectDB, and Doris

     Parameter	Description
     JDBC URL	The JDBC URL configured in the selected cluster. This field is read-only.
     catalog	You can select default catalog or external catalog. default catalog: Manages internal data in the cluster. external catalog: You can select from all external catalogs in the cluster or manually enter a catalog.
     database	You can select from all databases under the selected catalog or enter a database name.
     authentication user	You can select Same as cluster or Custom. Same as cluster is the default. If you select Custom, you must also enter the authentication username and password. Ensure the user has the required data permissions for tasks to run properly.
     task resource group	Resource groups isolate resources, and classifiers are used to map tasks to resource groups. Dataphin lets you specify resource group names for tasks of different priorities. When a task runs, its resource group is automatically set to manage resource allocation and isolation. Use default execution user: Uses the default execution user configured in the selected cluster. Custom: Includes five priority levels: Highest, High, Medium, Low, and Lowest. You must enter a resource group name for each priority level.

     Lindorm (compute engine)

     • Lindorm compute engine configuration

       Parameter	Description
       JDBC URL	Configure the JDBC URL for Lindorm (compute engine). To obtain the address, see View connection addresses.
       database	Enter the database name for Lindorm (compute engine).

     • Lindorm resource group settings

       The Lindorm resource group settings in this mode are the same as in single-engine mode. For more information, see Lindorm resource group settings.

     GaussDB (DWS) and Hologres

     Parameter	Description
     JDBC URL	Defaults to the JDBC URL of the selected cluster and cannot be modified.
     Schema	Enter the schema connection information for the compute engine database.

     Databricks

     In multi-engine mode, the compute configuration for a Databricks compute source is the same as in single-engine mode. For more information, see Compute configuration.

     Amazon EMR

     An Amazon EMR compute source has the same compute configuration in multi-engine mode and single-engine mode. For more information, see Compute configuration.

     Alibaba Cloud EMR Serverless Spark

     Parameter		Description
     Connection Configuration	Workspace	The workspace selected in the chosen cluster. This parameter cannot be modified.
     	Catalog	Select a catalog. The options include all catalogs in the selected cluster.
     	Schema	Select a schema. The options include all schemas in the selected catalog.
     Gateway Configuration	Production Task Default Gateway	Select a gateway. The options include all gateways that have been added to the selected cluster. Hover over a gateway name to view its status, specifications, Spark engine version, Kyuubi Service engine version, queue name, access method, and number of high-availability replicas.
     	Other Tasks Gateway
     	Priority Task Gateway	By default, the production task default gateway is selected. If you select Custom, you must also select a gateway for each priority level.

     Transwarp TDH 6.x, Transwarp TDH 9.3.x, and Transwarp ArgoDB

     Parameter	Description
     JDBC URL	The JDBC URL for the selected cluster. This setting cannot be modified.
     Database	Select a database. The options include all databases within the selected cluster.
     Execution user	Select Use Cluster Configuration or Custom Configuration. If you select Use Cluster Configuration, the execution user field displays the configuration from the cluster. If you select Custom Configuration, you can manually enter a username.
     Execution user for development tasks	The username for executing tasks in the development environment.
     Execution user for scheduled tasks	The username for executing scheduled tasks.
     Priority task queue	Select an option for the priority execution user: Use default execution user or Custom. If you select Custom, configure the usernames for tasks with different priorities. Note Priority queues allocate resources by creating different YARN queues on a Hadoop cluster. Tasks are sent to the YARN queue that corresponds to their priority level for execution.

     Flink, Ververica Flink, and FusionInsight Flink

     The configuration for Flink, Ververica Flink, and FusionInsight Flink compute sources is the same in multi-engine mode as it is in single-engine mode. For more information, see Create a Flink compute source, Create a Ververica Flink compute source, and Create a FusionInsight Flink compute source.

3. Click Test Connection to test the connection to the compute source.

4. After the connection test is successful, click Submit.