Train an EasyRec model in PAI-DLC

1. Go to Alibaba Cloud Platform for AI (PAI) Distributed Training (DLC)

Log on to the PAI console. In the left-side navigation pane, choose Model Training > Distributed Training (DLC) to go to the Jobs page.

2. Create a job

• Enter the image address.

  Note: The 0.8.5 tag indicates the EasyRec version.

  Note that DeepRec is not supported.

• Mount OSS.

  Mount an OSS bucket path to DLC to store the training model and configuration files. If your training data is in OSS, you must retrieve it from there. If your training data is in MaxCompute, you can retrieve it directly from there.

• Job command

  Example

  cat  << EOF >> odps_conf
  access_id=****************
  access_key=****************
  end_point=http://service.cn-hangzhou.maxcompute.aliyun.com/api
  
  EOF
  
  export ODPS_CONFIG_FILE_PATH=odps_conf
  python -m easy_rec.python.train_eval --pipeline_config_path /mnt/data/dwd_avazu_ctr_deepmodel_ext.config --continue_train --train_input_path odps://pai_online_project/tables/dwd_avazu_ctr_deepmodel_train --eval_input_path odps://pai_online_project/tables/dwd_avazu_ctr_deepmodel_test --model_dir /mnt/data/dlc_demo/ --edit_config_json='{"data_config.num_epochs":1, "data_config.input_type":"OdpsInputV3"}'

• Set job resources.

  Select appropriate resources based on the data volume and model size. You can determine the number of cores based on the number of CUs used for training in MaxCompute, or refer to the following table for an estimate.

  Sample Size	Feature Count	PS Count	PS Cores	Workers Count	Workers Cores
  100,000	100	1	4	2	4
  100,000	1,000	1	4	4	4
  1,000,000	100	1	8	3	8
  1,000,000	1,000	1	16	9	8
  10,000,000	100	1	12	4	8
  10,000,000	1,000	2	16	12	8

3. Train the model

After you complete the configuration, click Submit to start the job.

4. Deploy the model

After the model is trained, you can import the DLC job into DataWorks for periodic scheduling.

Step 1: Create a DLC job

Log on to the PAI console, go to the Distributed Training Job page, and create a DLC job.

Step 2: Create a PAI DLC node

1. Log on to the DataWorks console. In the left-side navigation pane, choose Data Development > Data Development. Select your workspace from the drop-down list and click Go to Data Development.

2. Right-click the target workflow and choose Create Node > Algorithm > PAI DLC.

3. In the Create Node dialog box, enter a Name for the node and click OK. You can then develop and configure the job on the node.

On the node editing page, select your DLC job from the drop-down list.

After the job is loaded, the DLC node editor generates the corresponding node code based on the job configuration in PAI. You can edit this code to modify the job. For more information, see Develop a PAI DLC job.

Alternatively, you can write the DLC job code directly:

dlc submit tfjob \
    --name=dlc_test_clone${train_ymd} \
    --command='
cat  << EOF >> odps_conf
access_id=****************
access_key=****************
end_point=http://service.cn-hangzhou.maxcompute.aliyun.com/api

EOF

export ODPS_CONFIG_FILE_PATH=odps_conf
python -m easy_rec.python.train_eval \
--pipeline_config_path /mnt/data/dwd_avazu_ctr_deepmodel_ext.config \
--continue_train --train_input_path odps://pai_online_project/tables/dwd_avazu_ctr_deepmodel_train \
--eval_input_path odps://pai_online_project/tables/dwd_avazu_ctr_deepmodel_test \
--model_dir /mnt/data/dlc_demox${train_ymd}/ \
--edit_config_json='\''{"data_config.input_type":"OdpsInputV3"}'\''' \
    --workspace_id=${pai_workspace_id} \
    --priority=1 \
    --workers=3 \
    --worker_image=mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easyrec/easyrec:py36-tf1.15-0.8.5 \
    --worker_spec=ecs.g6.2xlarge \
    --ps=1 \
    --ps_image=mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easyrec/easyrec:py36-tf1.15-0.8.5 \
    --ps_spec=ecs.g6.2xlarge \
    --chief=true \
    --chief_image=mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easyrec/easyrec:py36-tf1.15-0.8.5 \
    --chief_spec=ecs.g6.large \
    --data_sources=${oss_sources_id}

4. Note

To avoid creating indistinguishable jobs when using periodic scheduling in DataWorks, append a date-time variable to the job name, such as ${train_ymd}. You can then assign a scheduling parameter to this variable. For more information, see Develop a PAI DLC job.

Step 3: Configure job scheduling

In the right-side pane of the node editing page, click Scheduling Configuration. The Scheduling Configuration panel includes tabs such as Basic Properties, Parameters, Time Properties, Resource Properties, and Scheduling Dependencies. On the Time Properties tab, you can configure the scheduling cycle. DataWorks then automatically runs the node job based on this configuration. For more information, see Overview of job scheduling properties.

Note

• You must set the rerun properties and the upstream dependencies for the node before you can submit it.

Step 4: Debug the job code

You can perform the following debugging operations to verify that the job runs as expected.

1. (Optional) Select a resource group for running the job and assign values to custom parameters.

   • Click the Run with Parameters (F6) icon in the toolbar. In the Parameters dialog box, select the scheduling resource group to use for debugging.

   • If your job code uses scheduling parameter variables, you can assign values to the variables here for debugging. For more information about the parameter assignment logic, see Scheduling Parameters.

2. Save and run the code.

   Click the save icon in the toolbar to save the code, and then click the run icon to run the job.

3. (Optional) Perform smoke testing.

   If you want to perform smoke testing in the development environment to check whether the scheduled node job runs as expected, you can do so when you submit the node or after it has been submitted. For more information, see Perform smoke testing.

Step 5: Submit and publish the job

After configuring the node job, you must submit and publish it. The node then runs periodically according to its scheduling configuration.

1. Click the save icon in the toolbar to save the node.

   Click the submit icon in the toolbar to submit the node job.

   When you submit the job, enter a Change Description in the Submit New Version dialog box and choose whether to perform a code review after submission.

2. Note

   • You must set the rerun properties and upstream dependencies for the node before you can submit it.

   • A code review improves code quality and helps prevent incorrect code from being published to the production environment. If you enable code review, a reviewer must approve the submitted code before it can be published. For more information, see Code review.

If you use a workspace in standard mode, click Publish in the upper-right corner after submitting the job to publish it to the production environment. For more information, see Publish a job.

Step 6: View O&M logs

Click O&M in the upper-right corner of the editing page to view the scheduled job's status in the O&M center. For more information, see Manage periodic jobs.