DocsICP FilingConsole
官方文档
首页

Model training

更新时间：
复制 MD 格式
产品详情
我的收藏

This component trains EasyRec models.

Prerequisites

You must activate Object Storage Service (OSS) and grant the required permissions. For more information, see Activate OSS and Cloud product dependencies and authorizations: Designer.

Component Configuration

You can configure the model training component in the following ways.

Configure the component visually

  • Input ports

Input port (from left to right)

Recommended upstream component

Corresponding PAI command parameter

Required

Negative sampling item feature table

Note

Negative sampling parameters are typically used in specific algorithms such as Deep Structured Semantic Models (DSSM).

data_config.negative_sampler.input_path

No

EasyRec configuration file

Note

The full OSS path of the configuration file is required.

config

No

Training table

train_tables

Yes

Evaluation table

eval_tables

Yes

fine_tune_checkpoint

Note

The model continues training based on this checkpoint.

  • Data type: Model stored in OSS

  • Supported components: Read OSS Data

train_config.fine_tune_checkpoint in edit_config_json

No

Binning table

boundary_table

No

  • Component parameters

Tab

Parameter

Required

Description

Corresponding PAI command parameter

Default value

Parameter Settings

Model path

No

The model storage path.

model_dir

Pipeline data path

EasyRec configuration file

No

If you do not provide a configuration file from the input port, you can refer to model_config, enter the configuration in the editor, and save it to a specified OSS path.

config

None

Select target columns for training and evaluation

No

This parameter is available only when you select Is RTP FG mode.

Specifies the target columns for training and evaluation.

Combined into the selected_cols parameter

None

Weight columns for training and evaluation

No

This parameter is available only when you select Is RTP FG mode.

Specifies the weight columns for training and evaluation.

None

Feature columns for training and evaluation

No

This parameter is available only when you select Is RTP FG mode.

Specifies the feature columns for training and evaluation.

None

Specify algorithm version

No

After you select Advanced Options, you can customize the EasyRec execution version.

  1. First, refer to EasyRec version updates to generate an EasyRec TAR package.

  2. Upload the TAR package of the corresponding version to an OSS path. For more information, see Upload a file in the console.

  3. Select the uploaded TAR file for this parameter.

script

Empty

Hyperparameter configuration edit_config_json

No

After you select Advanced Options, enter the content that you want to add to the EasyRec configuration file in the hyperparameter configuration. The component adds the hyperparameter configuration to the EasyRec configuration file.

edit_config_json

None

Execution tuning

Number of PS

No

The number of parameter server (PS) nodes.

The complete execution tuning parameters are combined into the cluster parameter

2

Number of PS CPUs

No

The number of CPUs requested by each PS. A value of 1 indicates one CPU core.

10

PS memory size (MB)

No

The memory requested by each PS. A value of 100 indicates 100 MB.

40000

Number of workers

No

The number of worker nodes.

6

Number of worker CPUs

No

The number of CPUs requested by each worker. A value of 1 indicates one CPU core.

8

Worker memory usage (MB)

No

The memory requested by each worker. A value of 100 indicates 100 MB.

40000

Number of worker GPUs

No

GPUs are generally not required for EasyRec training.

0

PAI command and description

PAI -project algo_public -name easy_rec_ext 
    -Darn="acs:ram::xxxx:role/aliyunodpspaidefaultrole" 
    -Dbuckets="oss://rec_sln_demo/" 
    -Dcluster="{\"ps\": {\"count\": 2, \"cpu\": 1000, \"memory\": 40000}, \"worker\": {\"count\": 6, \"cpu\": 800, \"gpu\": 0, \"memory\": 40000}}" 
    -Dcmd="train" 
    -Dconfig="oss://rec_sln_demo/EasyRec/deploy/rec_sln_demo_dssm_recall_v1/rec_sln_demo_dssm_recall_v1.config" 
    -Deval_tables="odps://pai_hangzhou/tables/pai_temp_flow_26un8zq7v4goadi373_node_39w13qw9osm9rdbu0h_outputTable" 
    -Dlifecycle="28" 
    -Dmodel_dir="oss://rec_sln_demo/EasyRec/deploy/rec_sln_demo_dssm_recall_v1/20230425" 
    -DossHost="oss-cn-hangzhou-internal.aliyuncs.com" 
    -Dscript="oss://rec_sln_demo/easy_rec_ext_0.6.1_res.tar.gz" 
    -Dselected_cols="is_click,features" 
    -Dtables="odps://pai_hangzhou/tables/pai_temp_flow_26un8zq7v4goadi373_node_4ijqwcg7upzteu5036_outputTable,odps://pai_hangzhou/tables/pai_temp_flow_26un8zq7v4goadi373_node_39w13qw9osm9rdbu0h_outputTable,odps://pai_hangzhou/tables/pai_temp_flow_fty24i21e9dzvzj6a0_node_svxd0bqu2x7ep8furu_outputTable" 
    -Dtrain_tables="odps://pai_hangzhou/tables/pai_temp_flow_26un8zq7v4goadi373_node_4ijqwcg7upzteu5036_outputTable"
    -Dedit_config_json="{\"train_config.fine_tune_checkpoint\": \"oss://rec_sln_demo/EasyRec/deploy/rec_sln_demo_dssm_recall_v1/20230405/\", \"data_config.negative_sampler.input_path\": \"odps://pai_hangzhou/tables/pai_temp_flow_fty24i21e9dzvzj6a0_node_svxd0bqu2x7ep8furu_outputTable\"}" ;

Parameter name

Required

Description

cmd

Yes

Set cmd to train to run model training.

config

Yes

The EasyRec configuration file for training. Specify the full OSS path of the configuration file.

train_tables

Yes

The training table. The format is odps://{project}/tables/{table_name}. Separate multiple training tables with a comma (,).

eval_tables

Yes

The evaluation table. The format is odps://{project}/tables/{table_name}. Separate multiple evaluation tables with a comma (,).

arn

Yes

Specifies the authorization of a resource group. Log on to the PAI console. On the Activation and Authorization>All Cloud Product Dependencies page, in the Designer section, click View Authorization Information in the Operation column to obtain the arn.

ossHost

Yes

The OSS endpoint for each region. For information about how to obtain an endpoint, see Endpoints and data centers.

buckets

Yes

The bucket where the configuration file is located and the bucket where the model is saved. If there are multiple buckets, separate them with a comma (,), for example, oss://xxxx/,oss://xxxx/.

model_dir

Yes

The model directory. If you specify model_dir, it overwrites the model_dir in the configuration file. This parameter is typically used for periodic scheduling.

edit_config_json

No

Modifies fields in the configuration file in JSON format. For example: edit_config_json="{\"train_config.fine_tune_checkpoint\": \"oss://xxx/\"}".

script

No

Specifies the EasyRec algorithm TAR file.

selected_cols

No

The columns in the table that are used for training and evaluation. This helps improve the training speed.

Usage example

  1. Prepare the following datasets:

    • train: pai_online_project.easyrec_demo_taobao_train_data

    • test: pai_online_project.easyrec_demo_taobao_test_data

    Note

    Ideally, you should use your own ODPS tables, but for testing purposes, the two provided tables are publicly accessible.

  2. Create a workflow as shown in the following figure.image..png

    Area

    Description

    ①

    Set the Table Name parameter of Read Data Table-1 to pai_online_project.easyrec_demo_taobao_train_data_training_table.

    ②

    Set the Table Name parameter of Read Data Table-2 to pai_online_project.easyrec_demo_taobao_test_data_test_table.

    ③

    • Upload the configuration file to OSS and select the file for the EasyRec configuration file parameter.

    • For the Model path parameter, select the model storage path.

  3. After the workflow finishes, you can view the output model file.

    You can view the output model file in the OSS path that you configured for the Model path parameter.

  4. View and analyze logs in Logview.

    When the EasyRec command runs, Designer provides a link to Logview. Right-click the model training component and click View Log in the shortcut menu. You can use Logview to quickly view the model training results or locate errors.

image..png

On the Worker running page, you can select a task instance and its corresponding worker instance to view the details.

image..png

Where:

  • worker 0 is a training worker. Workers 2 to 7 are also training workers. Click the icon in the StdErr column to view the training process.

  • worker 1 is an evaluation worker. Click the icon in the StdErr column to view the model's metrics on the evaluation set.

For more usage examples, see the 8_rec_sln_demo_rec_sln_demo_sorting_v2_train node in the Recommendation Algorithm Customization - Sorting example and the 12_rec_sln_demo_dssm_recall_v1_train node in the Recommendation Algorithm Customization - Vector Recall example.

Previous：Component reference: EasyRecNext：Model prediction