Associate a Lindorm computing resource

Limitations

• Regions: China (Hangzhou), China (Shanghai), China (Beijing), China (Ulanqab), China (Shenzhen), China (Chengdu), China (Hong Kong), Japan (Tokyo), Singapore, Malaysia (Kuala Lumpur), and Indonesia (Jakarta).

• Permissions:

  • You must use a DataWorks Serverless resource group to run Lindorm tasks in DataWorks.

  • Only workspace members with the O&M or Workspace Administrator role, or those with the AliyunDataWorksFullAccess permission, can create computing resources. To view member roles or grant permissions, see Add workspace members and manage member roles and permissions.

Go to the Computing Resources page

1. Go to the DataWorks Workspaces page. In the top navigation bar, select the target region and find the workspace where you want to create a computing resource. Click the workspace name or Details in the Operation column to go to the workspace details page.

2. In the left-side navigation pane, click Computing Resources.

Associate a Lindorm computing resource

On the Computing Resources page, configure the parameters to associate the Lindorm computing resource.

1. Select the type of computing resource to associate.

   1. Click Associate Computing Resources to open the Associate Computing Resources page.

   2. On the Associate Computing Resources page, select Lindorm as the computing resource type to go to the Bind Lindorm Computing Resource configuration page.

2. Configure the Lindorm computing resource.

   On the Associate Lindorm Computing Resource page, configure the following parameters.

   Section	Parameter	Description
   Basic Information	Configuration Mode	Only ApsaraDB for RDS is supported.
   	Instance	Select the Lindorm instance to associate with DataWorks. The drop-down list displays Lindorm instances in the current region.
   	Database Name	Select the default database for this computing resource. DataWorks connects to this database when running tasks. The default value is default.
   	Username/Password	Enter the username and password for DataWorks to authenticate with this Lindorm computing resource. To obtain credentials, go to the Lindorm console, click the Instance Name, and find the credentials in the Database Connection section of the left-side navigation pane.
   	Computing Resource Instance Name	Use this name to identify and select the resource when configuring tasks.
   Connection Configuration	Connected state	In the connection configuration section, select the Serverless resource group that DataWorks uses to run Lindorm tasks and click Test Connectivity to verify that the resource group can access your Lindorm instance. For more information, see Network connectivity solutions.

3. Click Confirm to complete the configuration of the Lindorm computing resource.

Background

In DataWorks, you can configure the Spark parameters that scheduling nodes use at runtime in the following ways:

Method 1: Configure global Spark parameters at the workspace level to control the Spark settings that each DataWorks module uses for Lindorm tasks. You can also define whether global parameters take precedence over module-specific ones. For more information, see Configure global Spark parameters.

Method 2: In Data Studio, set Spark properties for individual node tasks on the node editing page. Other modules do not support per-task Spark property configuration.

Access control

Only the following roles can configure global Spark parameters:

• Alibaba Cloud account.

• A RAM user or RAM role with the AliyunDataWorksFullAccess permission.

• A RAM user with the Workspace Administrator role.

View global Spark parameters

1. Go to the Computing Resources page and find your associated Lindorm computing resource.

2. Click Spark Parameters to open the Spark parameter configuration pane.

Configure global Spark parameters

Follow these steps to configure global Spark parameters. For details about specific parameters, see Job configuration.

1. Go to the Computing Resources page and find your associated Lindorm computing resource.

2. Click Spark Parameters to open the Spark parameter configuration pane.

3. Set global Spark parameters.

   Click Edit Spark Parameters in the upper-right corner of the Spark Parameters pane to configure the global Spark parameters and priorities for each module.

   Note

   This is a workspace-level setting. Ensure you are in the correct workspace before proceeding.

   Parameter	Steps
   Spark Property	Configure the Spark properties that each module uses when it runs Lindorm tasks. For more information, see Job configuration. Click Add and enter the Spark Property Name and the corresponding Spark Property Value. Note To enable data lineage and output collection, configure the following properties: Set Spark Property Name to spark.sql.queryExecutionListeners. Set Spark Property Value to com.aliyun.dataworks.meta.lineage.LineageListener. For more information about Spark property settings, see Job configuration.
   Global Settings Take Precedence	If you select this option, global configurations take precedence over configurations within product modules. In this case, tasks are run based on the globally configured Spark properties. Global configuration: The Spark properties set on the Spark Parameters page for the Lindorm computing resource in Management Center > Computing Resources. Currently, you can set global Spark parameters only for the Data Studio and Operation Center modules. Configuration within product modules: Data Studio: For Lindorm Spark and Lindorm Spark SQL nodes, you can set Spark properties for a single node task on the Configuration Items tab of the node editing page. Other product modules: Setting Spark properties separately within a module is not supported.

4. Click Confirm to save the global Spark parameters.