The Update EasyRec Config component runs the add_feature_info_to_config.py script from EasyRec to merge feature statistics into the template.config file generated by PAIREC.
Prerequisites
Before you begin, make sure you have:
Object Storage Service (OSS) activated
Machine Learning Designer authorized to access OSS. For details, see Activate OSS and Grant permissions for Designer
Configure the component
You can configure the Update EasyRec Config component using the PAI console or PAI commands.
Method 1: PAI console
Input port
| Input port | Description | Recommended upstream component | PAI command parameter | Required |
|---|---|---|---|---|
| config table input | A statistical table that records the number of occurrences of each feature and the bucket values of numeric features | MaxCompute tables, via SQL Scripts or Read table | config_table | Yes |
Parameters Setting tab
| Parameter | Required | Description | PAI command parameter | Default |
|---|---|---|---|---|
| rectemplate produce template.config | Yes | The OSS path where the template configuration file generated by PAIREC is stored | template_config_path | — |
| easyrec.config output path | Yes | The OSS directory where the generated EasyRec configuration file is saved. Select a directory within a bucket, not the bucket root. | output_config_path | — |
| easyrec.config filename | Yes | The file name for the generated EasyRec configuration file. | — | — |
| Specify the algorithm version | Yes | The OSS path of the EasyRec TAR package. To prepare the package: (1) Build or download the TAR package from Release & Upgrade. (2) Upload it to OSS. (3) Select the uploaded path. | script | — |
Model Tuning tab
All parameters on this tab are passed to EasyRec as the cluster parameter.
| Parameter | Required | Description | Default |
|---|---|---|---|
| Worker Count | No | Number of worker nodes | 1 |
| Worker CPU | No | vCPUs per worker node | 8 |
| Worker Memory | No | Memory per worker node, in MB | 40000 |
| Worker GPU | No | GPUs per worker node. GPUs are not required for most EasyRec training jobs. | 0 |
Output port
| Output port | Data type | PAI command parameter | Required |
|---|---|---|---|
| easyrec config output | OSS path | output_config_path | Yes |
Connect this output to the model training component.
Method 2: PAI commands
Run the following command in the SQL Script component:
PAI -project algo_public -name easy_rec_ext
-Darn="acs:ram::xxx:role/aliyunodpspaidefaultrole"
-Dbuckets="oss://rec_sln_demo/"
-Dcluster="{\"worker\": {\"count\": 1, \"cpu\": 800, \"gpu\": 0, \"memory\": 40000}}"
-Dcmd="custom"
-DentryFile="easy_rec/python/tools/add_feature_info_to_config.py"
-Dextra_params="--template_config_path=oss://rec_sln_demo/EasyRec/deploy/rec_sln_demo_dssm_recall_v1/rec_sln_demo_dssm_recall_v1_template.config --output_config_path=oss://rec_sln_demo/EasyRec/deploy/rec_sln_demo_dssm_recall_v1//rec_sln_demo_dssm_recall_v1.config --config_table=odps://pai_hangzhou/tables/pai_temp_flow_26un8zq7v4goadi373_node_2m6yfr7q3a69m9jv7n_outputTable"
-Dlifecycle="28"
-DossHost="oss-cn-hangzhou-internal.aliyuncs.com"
-Dscript="oss://rec_sln_demo/easy_rec_ext_0.6.1_res.tar.gz"
-Dtables="odps://pai_hangzhou/tables/pai_temp_flow_26un8zq7v4goadi373_node_2m6yfr7q3a69m9jv7n_outputTable";Parameter reference
| Parameter | Required | Description |
|---|---|---|
entryFile | Yes | Entry script. Always set to easy_rec/python/tools/add_feature_info_to_config.py. |
cmd | Yes | Set to custom to run the custom EasyRec script. |
arn | Yes | Alibaba Cloud Resource Name (ARN) for resource group authorization. To get the ARN: log on to the PAI console, choose Activation & Authorization > Dependent Services, and click View Authorization in the Designer section. |
ossHost | Yes | OSS endpoint for the region. For a list of endpoints, see Regions and endpoints. |
buckets | Yes | OSS bucket where the EasyRec TAR package and model are stored. Separate multiple buckets with commas. Example: oss://bucket-a/,oss://bucket-b/. |
extra_params | Yes | Additional parameters passed to the entry script: --template_config_path (PAIREC-generated template), --output_config_path (output path), and --config_table (feature statistics table from MaxCompute). |
script | No | OSS path of the EasyRec TAR package. Build or download the package from Release & Upgrade, upload it to OSS, and provide the path here. Sample package: easy_rec_ext_0.6.1_res.tar.gz. |
Example
This example walks through configuring the Update EasyRec Config component using sample DSSM recall data.
Download the sample files:
Feature information: dssm_recall_30d_config_v1.csv
Template configuration: template.config
These files are generated by PAIREC. They are provided here to help you test the component without running PAIREC first.
Create a MaxCompute table for the feature statistics. For setup instructions, see Connect using the local client (odpscmd).
CREATE TABLE IF NOT EXISTS dssm_recall_30d_config_v1(feature STRING, feature_info STRING, message STRING);Upload the CSV file to the MaxCompute table using Tunnel. For details, see Tunnel command.
tunnel upload dssm_recall_30d_config_v1.csv dssm_recall_30d_config_v1 -fd \t;Upload the
template.configfile to OSS. For details, see Upload objects.Create a pipeline in Machine Learning Designer. The pipeline looks like the following: Configure the components as follows:
Section Configuration 1 Set the Table Name parameter of the Read Table-51 component to dssm_recall_30d_config_v1.2 On the Parameters Setting tab of the Update Easyrec Config-1 component, set the following parameters: rectemplate produce template.config: Select the OSS path where you uploaded template.config. easyrec.config output path: Select a directory within an OSS bucket as the output path. You cannot select the bucket root. easyrec.config filename: Enter a custom file name. Specify the algorithm version: Select the OSS path of the EasyRec TAR package. See Release & Upgrade of EasyRec documentation. Sample package: easy_rec_ext_0.6.1_res.tar.gz.
Click the
icon to run the pipeline.
After the pipeline completes, the EasyRec configuration file is saved to the OSS directory specified in easyrec.config output path.
What's next
The Update EasyRec Config component is used in the _11_rec_sln_demo_dssm_recall_v1_update_config_ node. For a complete end-to-end walkthrough, see DSSM vector recall.