Create and use a PAI DLC node

Prerequisites

• Grant DataWorks access to PAI.

  You can go to the Authorization Page to grant the permissions with a single click. For more information about the permission policy, see AliyunServiceRoleForDataWorksEngine. Only an Alibaba Cloud account or a RAM user with the AliyunDataWorksFullAccess policy can perform this one-click authorization.

• Create a workflow.

  You must create a workflow before creating a node. For more information, see Create a workflow.

Considerations

• Each completed PAI DLC node run generates a new DLC task on the PAI Distributed Training (DLC) platform. To avoid accumulating many identically named tasks, set a reasonable scheduling frequency and add date and time variables to the task name. Assign time-based scheduling parameters to these variables so that each task name includes a unique timestamp. For more information, see Step 2: Develop a PAI DLC task.

• DataWorks does not support running PAI DLC tasks on the public scheduling resource group.

Step 1: Create a PAI DLC node

1. Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.

2. Right-click the target workflow and choose Create Node > Machine Learning > PAI DLC.

3. In the Create Node dialog box, enter a Name for the node and click OK. You can then develop and configure the task within the node.

Step 2: Develop a PAI DLC task

Task code example

In the PAI DLC node editor, you can write a DLC task in the following ways:

• Generate code from an existing DLC task.

  Search for an existing DLC task by name and load it. The node editor generates corresponding code based on the task's PAI configuration. You can then edit this code as needed.

  Note

  • If you do not have permission to load or create a task, follow the on-screen instructions to grant authorization.

  • If no tasks are available, create one in the PAI console. You can create PAI DLC tasks in multiple ways based on your needs. For details, see Create a training task, Create a training task: Python SDK, and Create a training task: Command line.

• Write the DLC task code directly.

  Write the task code directly in the PAI DLC node editor.

After you run the PAI DLC node, a new DLC task is generated in PAI based on the configuration. The following code provides an example:

dlc submit xgboostjob \   #Submits the DLC task.
    --name=wsytest_pai04_XGBoost \   #The name of the DLC task. We recommend using a variable name or the DataWorks node name.
    --command='echo '\''${variable_name}'\'';' \   #The command to be executed by the DLC task.
    --workspace_id=80593 \   #The ID of the workspace where the DLC task runs.
    --priority=1 \   #The task priority. Valid values: 1 to 9. 1 is the lowest priority and 9 is the highest.
    --workers=1 \    #The number of workers. If the value is greater than 1, the task is distributed and can run in parallel on multiple workers.
    --worker_image=registry.cn-hangzhou.aliyuncs.com/pai-dlc/tensorflow-training:2.3-cpu-py36-ubuntu18.04 \   #The worker image that provides the runtime environment for the DLC task.
    --worker_spec=ecs.g6.xlarge   #The worker specification, which is the compute instance type used by the workers.

Use scheduling parameters in code

DataWorks provides Scheduling Parameter to dynamically pass variables to your code. Define variables in your task code using the ${variable_name} format and assign values in the Scheduling > Parameter section. For more information about supported formats, see Supported formats for scheduling parameters.

The following code shows an example of using a scheduling parameter:

--command='echo '\''${variable_name}'\'';' \ -- Can be used with scheduling parameters.

Step 3: Configure task scheduling

To run the node task periodically, click Scheduling in the right-side pane and configure the scheduling properties. For more information, see Overview of task scheduling properties.

Step 5: Submit and deploy the task

After the node task is configured, submit and deploy it. After deployment, the node runs periodically based on its scheduling configuration.

1. Click the icon in the toolbar to save the node.

2. Click the icon in the toolbar to submit the node task.

   In the Commit Node dialog box, enter a Change Description. You can also choose whether to perform a code review after submission.

   Note

   • You must set the Rerun attribute and Parent Nodes for the node before you can submit the node.

   • Code review helps ensure code quality and prevents erroneous tasks from being deployed to production. If you enable code review, the submitted code must be approved before deployment. For more information, see Code review.

If you use a workspace in standard mode, click Deploy in the upper-right corner of the node editing page after submission to deploy the task to production. For more information, see Deploy a task.

Next steps

After the task is submitted and deployed, it runs periodically based on the node's configuration. Click O&M Personnel in the upper-right corner of the editor page to view the scheduling and execution status. For more information, see Manage scheduled tasks.