How to configure a Lindorm Spark node-Data Management-阿里云帮助中心

The Lindorm Spark node for task orchestration uses the Lindorm compute engine to perform distributed computing tasks efficiently. It is ideal for scenarios such as data production, interactive analysis, machine learning, and graph computing. You can also read data from LindormTable and jointly analyze it with data in OSS.

Prerequisites

You have an Alibaba Cloud account or a Resource Access Management (RAM) user with the AliyunLindormFullAccess access policy.
The compute engine is enabled for the Lindorm instance. For more information, see Enable services.
Note
Enable the History Server feature for the Lindorm instance in the Lindorm console to view historical operational logs of tasks.

Background information

The Lindorm compute engine is a high-performance, low-cost, stable, and reliable distributed computing service built on the core capabilities of Lindorm. For more information about the Lindorm compute engine, see Engine overview.

Billing

The compute engine for a Lindorm instance is billed based on the actual usage of computing resources. You are charged only for the resources that you use. For more information, see Billing methods.

Procedure

Log on to the DMS console V5.0.
In the top navigation bar, choose Data+AI > Data+AI > Task Orchestration.

Note
If you use the DMS console in simple mode, move the pointer over the icon in the upper-left corner and choose All Features > Data+AI > Data Development > Task Orchestration.
Click the name of the target task flow to open its details page.
Note
To create a task flow, see Create a task flow.
From the Task Type list on the left side of the canvas, drag the Lindorm Spark node to a blank area on the canvas.
Double-click the Lindorm Spark node.
In the Basic Configuration section, configure the basic information for the node.
Configuration item
Description
Region
Select the region where the Lindorm instance is located.
Lindorm instance
Select the Lindorm instance.
Task type
Select the task type:
JAR
Python
SQL

In the Job Configuration section, configure the JSON request parameters.

Basic job parameters:

Parameter	Description	Example
mainResource	The path in OSS where the job file (.jar, .py, or .sql) is stored.	oss://bucket/path/spark-examples.jar
mainClass	The entry class path of the main function in the .jar job file. Note This parameter is not required if the task type is Python or SQL.	com.aliyun.ldspark.SparkPi
args (optional)	The arguments passed to the mainClass parameter. Note This parameter is not required if the task type is SQL.	["arg1", "arg2.1, arg2.2"]
configs	Other configurations for the job.	{"spark.hadoop.fs.oss.impl":"org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem"}

configs parameters:

Parameter	Description	Example value
spark.hadoop.fs.oss.endpoint	The endpoint of the OSS where the job file is located. Note To view OSS endpoints, see Regions and endpoints.	oss-cn-beijing-internal.aliyuncs.com
spark.hadoop.fs.oss.accessKeyId	The AccessKey ID of your Alibaba Cloud account. It is used to identify the user. Note To obtain an AccessKey, see Create an AccessKey.	*******
spark.hadoop.fs.oss.accessKeySecret	The AccessKey secret of your Alibaba Cloud account. It is used to authenticate the user. Note To obtain an AccessKey, see Create an AccessKey.	*******
spark.hadoop.fs.oss.impl	The class used to access OSS.	Static field: org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
spark.sql.shuffle.partitions	The default number of partitions for shuffle operations in the Spark SQL engine.	20

If the task type is Python, configure the following Python job environment parameters in the configs parameter:
Parameter
Description
Example
spark.archives
The path of the runtime environment.
oss://OSS_BUCKET/pyspark_conda_env.tar.gz
spark.submit.pyFiles
The path of the Python file.
oss://OSS_BUCKET/your_project.zip
Example:
```
{
"spark.archives":"oss://OSS_BUCKET/pyspark_conda_env.tar.gz",
"spark.submit.pyFiles":"oss://OSS_BUCKET/your_project.zip"
}
```

(Optional) To access LindormTable, configure the following parameters in the configs parameter:

Parameter	Description	Example
spark.sql.catalog.Lindorm_table	The implementation class for accessing LindormTable. Note This parameter is not required if the task type is SQL.	Static field: com.alibaba.Lindorm.ldspark.datasources.v2.LindormCatalog
spark.sql.catalog.Lindorm_table.url	The endpoint used to access LindormTable over a virtual private cloud (VPC). Note This parameter is not required if the task type is SQL.	ld-bp1z350********-proxy-Lindorm.Lindorm.rds.aliyuncs.com:30020
spark.sql.catalog.Lindorm_table.username	The username for accessing LindormTable.	The default username is root.
spark.sql.catalog.Lindorm_table.password	The password for accessing LindormTable.	The default password is root.

Example:

{
"spark.sql.catalog.Lindorm_table.username": "my_user",
"spark.sql.catalog.Lindorm_table.password": "my_password",
}

Note

For more information about JAR tasks, see JAR job development practice.
For more information about Python tasks, see Python job development practice.

At the top of the page, click Trial Run.
- If the last line of the execution log is status SUCCEEDED, the trial run is successful.
- If the last line of the execution log is status FAILED, the trial run failed.
  Click Operational Log to view the reason for the failure. Modify the configuration and try again.
Note
After the task runs, you can view the operational logs for the compute engine job in the Lindorm console. For more information, see View the operational status of a job.

Configuration item	Description
Region	Select the region where the Lindorm instance is located.
Lindorm instance	Select the Lindorm instance.
Task type	Select the task type: JAR Python SQL