The AnalyticDB for MySQL Spark node in the task orchestration feature of Data Management (DMS) lets you schedule Spark jobs periodically and sequentially. This topic describes how to configure an AnalyticDB for MySQL Spark node.
Background information
When you develop AnalyticDB for MySQL Spark jobs using conventional tools, such as a Spark development editor or the command line, you can schedule jobs only one at a time. These jobs cannot be configured with dependencies or a specific execution order.
To resolve these issues, you can use the AnalyticDB for MySQL Spark node in DMS to develop and schedule your Spark jobs.
Scenarios
Machine learning
Image editing
Recommendation system
Report analysis
Data mining
Prerequisites
You have an AnalyticDB for MySQL Data Lakehouse Edition (V3.0) cluster. For more information, see Create a Data Lakehouse Edition cluster.
NoteThe cluster must be in a region supported by the AnalyticDB for MySQL Spark node. The supported regions are China (Zhangjiakou), China (Hangzhou), China (Shanghai), China (Shenzhen), China (Beijing), China (Hong Kong), US (Silicon Valley), US (Virginia), and Singapore.
You have created a resource group of the required type in the cluster. For more information, see Create and manage a resource group.
NoteIf you want to use the data warehouse feature of the AnalyticDB for MySQL Spark node, the following conditions must be met:
The cluster is in the China (Shanghai) region.
The task type of the resource group is Interactive.
The engine is set to Spark when you create the resource group.
If a Resource Access Management (RAM) user uses the AnalyticDB for MySQL Spark node, the associated Alibaba Cloud account must grant the
adb:SubmitSparkApp,adb:DescribeDBClusters, andadb:DescribeDBResourceGrouppermissions to the RAM user. For more information, see Manage the permissions of a RAM user.
Procedure
- Log on to the DMS console V5.0.
-
Move the pointer over the
icon in the upper-left corner and choose . NoteIf you use the DMS console in normal mode, choose in the top navigation bar.
Click the name of the target task flow to open its details page.
NoteTo create a task flow, see Create a task flow.
In the Task Type list on the left of the canvas, drag the ADB Spark node to the canvas.
On the node configuration page, click the Variable Settings tab on the right to configure variables.
NoteYou can click the
icon in the upper-right corner of the Variable Settings section to view tips on configuring variables.Click the Node Variable tab to configure node variables. For more information, see Configure time variables.
Click the Task Flow Constants tab to configure task flow constants. Task flow constants are static fields that can be used in all nodes. Use the
${name}format to reference them in SQL statements.Click the Task Flow Variable tab to configure task flow variables. For more information, see Configure time variables.
Click the Input Variables tab to view input variables.
Double-click the ADB Spark node and configure the following information:
Category
Configuration Item
Description
Basic Configurations
Region
Select the region where the target AnalyticDB for MySQL instance resides. Only China (Zhangjiakou), China (Hangzhou), and China (Shanghai) are supported.
ADB Instance
Select an existing instance. If you do not have an instance, click Create. For more information, see Create a Data Lakehouse Edition (V3.0) cluster.
ADB Resource Group
Select the target resource group. If the created resource group does not appear in the drop-down list, click Refresh. For more information, see Create and manage a resource group.
Task Type
Select Batch or SQL based on the job configuration.
NoteIf the resource group of the cluster is of the Interactive type and the engine is Spark, you can only set Task Type to SQL.
The task types are described as follows:
Batch: If you select this option, you can enter the JSON that describes the Spark job in the job configuration section.
SQL: If you select this option, you can enter SQL statements in the job configuration section.
Task Name
The name of the task in Spark. If you do not specify a name, the name of the task node (ADB Spark node) is used by default.
Job Configuration
-
Based on the selected Task Type, write the JSON or SQL statements in this section. For more information about how to configure Batch and SQL jobs, see Develop offline Spark applications and Develop Spark SQL applications.
After you complete the configuration, click Save.
Click Trial Run, Run at a Specific Time, or Run Within a Specific Time Range.
If the last line of the execution log is
status SUCCEEDED, the task ran successfully.If the last line of the execution log is
status FAILED, the task failed.NoteIf the task fails, check the execution log for the failed node and the cause of the failure. Then, modify the configuration and run the task again.
Configure the scheduling cycle.
Below the task type list, click the Task Flow Information tab.
In the Scheduling Configuration section, turn on the Enable Scheduling switch and configure the scheduling settings. For more information, see Overview of task orchestration.
Optional: Publish or unpublish the task flow. For more information, see Publish or unpublish a task flow.
Other operations
After a task has run, click Go to O&M in the upper-right corner of the page to go to the Operation Center. In the Operation Center, you can view task flow details, such as Creation Time, Creator or Owner, and Published status. You can also view the task's Execution Status (Succeeded, Failed, or Running) and its Start and End Time. On this page, you can perform O&M operations, such as pausing or rerunning the task.