Vector recall split model
Updated at:
The Vector Recall Split Model component uses the split_model_pai.py script to split the vector recall model.
Prerequisites
Before you begin, ensure that you have:
Object Storage Service (OSS) activated, with Machine Learning Designer authorized to access OSS. See Activate OSS and Grant the permissions that are required to use Machine Learning Designer
Completed model training
Configure the component
Input port
| Input port (left to right) | Data type / upstream component | PAI command parameter | Required |
|---|---|---|---|
| Model training path | OSS path / model training | model_dir | Yes |
Component parameters
Parameters Setting tab
| Parameter | Required | Description | PAI command parameter | Default value |
|---|---|---|---|---|
| exporter type | No | The export type of the trained model. Valid values: best, final, latest. Set via exporter_type:"best" in the config file; no PAI command parameter is used directly. | — | final |
| user model output path | No | The OSS output path for the user model. If left blank, Designer generates a random path under the pipeline data path. To set the pipeline data path, click the blank canvas and configure Pipeline data path on the Workflow properties tab. | user_model_dir | — |
| item model output path | No | The OSS output path for the item model. If left blank, the system generates a random path in the pipeline data directory. To specify the pipeline storage path, click the blank canvas and configure Data Storage on the Pipeline Attributes tab. | item_model_dir | — |
| user_fg.json input file | Yes | The OSS path of the user_fg.json feature generation configuration file. | user_fg_json_path | — |
| item_fg.json input file | Yes | The OSS path of the item_fg.json feature generation configuration file. | item_fg_json_path | — |
| Specify the algorithm version | Yes | The OSS path of a custom EasyRec TAR package. To use a specific EasyRec version: 1. Generate a TAR package following the EasyRec release notes. 2. Upload the TAR package to OSS. See Upload objects. 3. Select the uploaded file for this parameter. | script | None |
Tuning tab
| Parameter | Required | Description | Default value |
|---|---|---|---|
| Worker Count | No | Number of worker nodes. | 1 |
| Worker CPU Usage | No | Number of vCPUs per worker. A value of 1 specifies one vCPU. | 8 |
| Worker Memory Usage (MB) | No | Memory per worker, in MB. A value of 100 specifies 100 MB. | 40000 |
| Worker GPU Count | No | Number of GPUs per worker. In most EasyRec trainings, GPUs are not required. | 0 |
Output ports
| Output port (left to right) | Data type | PAI command parameter | Required |
|---|---|---|---|
| User model path | OSS path | user_model_dir | Yes |
| Path to the item model | OSS path | item_model_dir | Yes |
PAI command and parameters
PAI -project algo_public -name easy_rec_ext
-Dcmd="custom"
-Darn="<arn>"
-Dbuckets="oss://<oss-bucket>/"
-Dcluster="{\"worker\": {\"count\": 1, \"cpu\": 800, \"gpu\": 0, \"memory\": 40000}}"
-DentryFile="easy_rec/python/tools/split_model_pai.py"
-Dextra_params="--model_dir=oss://<oss-bucket>/EasyRec/deploy/<model-name>/<date>/export/final
--user_model_dir=oss://<oss-bucket>/EasyRec/deploy/<model-name>/<date>/export/final/user
--item_model_dir=oss://<oss-bucket>/EasyRec/deploy/<model-name>/<date>/export/final/item
--user_fg_json_path=oss://<oss-bucket>/EasyRec/deploy/<model-name>/user_fg.json
--item_fg_json_path=oss://<oss-bucket>/EasyRec/deploy/<model-name>/item_fg.json"
-Dlifecycle="28"
-DossHost="oss-<region>-internal.aliyuncs.com"
-Dscript="oss://<oss-bucket>/<easyrec-version>.tar.gz";Replace the placeholders with your actual values:
| Placeholder | Description | Example |
|---|---|---|
<arn> | Resource authorization information. To get this value, go to the PAI console and choose Activation & Authorization > Dependent Services. In the Designer section, click View Authorization in the Actions column. | acs:ram::123456789:role/aliyunodpspaidefaultrole |
<oss-bucket> | The OSS bucket that stores the configuration file and model outputs. If you use multiple buckets, separate them with commas: oss://bucket1/,oss://bucket2/. | my-rec-bucket |
<model-name> | The name of your trained model. | rec_sln_demo_dssm_recall_v1 |
<date> | The training date directory created during model export. | 20230425 |
<region> | The region where your OSS bucket is located. For a list of OSS endpoints by region, see Endpoints and data centers. | cn-hangzhou |
<easyrec-version> | The filename of your EasyRec TAR package in OSS. | easy_rec_ext_0.6.1_res.tar.gz |
Parameters
| Parameter | Required | Description |
|---|---|---|
entryFile | Yes | The entry script. Set to easy_rec/python/tools/split_model_pai.py. |
cmd | Yes | The command type. Set to custom. |
arn | Yes | Resource authorization information for the PAI job. |
ossHost | Yes | The internal OSS endpoint for the region where your bucket is located. |
buckets | Yes | All OSS buckets accessed by the job (configuration file and model output). Separate multiple buckets with commas. |
extra_params | Yes | Additional parameters that are not specified in the pipeline, including model_dir, user_model_dir, item_model_dir, user_fg_json_path, and item_fg_json_path. |
Example
For a complete end-to-end example using DSSM vector recall, see 12_rec_sln_demo_dssm_recall_v1_train in DSSM vector recall.
Is this page helpful?