Create an AutoML experiment in the console -Platform For AI(PAI)-阿里云帮助中心

Background information

The working mechanism of AutoML:

An experiment automatically generates hyperparameter combinations based on the algorithm configuration. Each hyperparameter combination corresponds to a Trial. Each Trial may correspond to a DLC job or one or more MaxCompute jobs, depending on the execution configuration of the experiment. The experiment schedules and runs multiple Trials, compares results, and finds an optimal hyperparameter combination. For more information about the working mechanism, see How AutoML works.

Prerequisites

Before you use AutoML for the first time, grant the required AutoML permissions.
A workspace is created.
To create a DLC job, complete the following preparations:
- Grant the required DLC permissions.
- Prepare a public or dedicated resource group for general computing or prepare a dedicated resource group for Lingjun resources (Lingjun Intelligent Computing).
To create a MaxCompute job, prepare MaxCompute resources and associate them with the workspace.

Procedure

Go to the AutoML page.
- Log on to the PAI console.
- In the left-side navigation pane, click Workspaces. On the Workspaces page, click the name of the workspace that you want to manage.
- In the left-side navigation pane, choose Model Training > AutoML.
On the experiment list page, click .
On the page for creating an experiment, configure the following parameters.
Click .

The experiment appears in the experiment list.

Basic information

Parameter	Description
Name	Configure the experiment name based on the inline prompt.
Description	A brief description of the experiment to distinguish it from other experiments.
Visibility	The visibility of the experiment. Valid values: : visible only to you and administrators in the workspace. : visible to all users in the workspace.

Execution configurations

Job Type is the execution environment for Trials. You can select or .

: runs DLC jobs for hyperparameter tuning. For more information about DLC jobs, see Create a training job.
: runs SQL commands or PAI commands from Designer algorithm components on MaxCompute compute resources for hyperparameter tuning. For more information about Designer algorithm components and the PAI commands supported by each component, see Overview of Designer components.

DLC

Parameter	Description
	Select a public resource group or a purchased dedicated resource group. For more information about how to prepare a resource group, see Create a resource group and purchase general computing resources and Create a resource quota.
	The framework type. Valid values: Tensorflow, PyTorch.
	Configure this parameter to the prepared dataset.
	The storage location of the job code file (code repository information). Configure this parameter to the prepared Code configuration. DLC downloads the code to the specified working directory, so you must have access to the code repository.
	The runtime image for worker nodes. The following runtime image types are supported: : various official images provided by Alibaba Cloud PAI, supporting different resource types, Python versions, and deep learning frameworks TensorFlow and PyTorch. For the image list, see Public images. : a custom image added to PAI. Before you select this option, you must first Custom images. : configure a Docker Registry image URL accessible in a public network environment for your custom image, community image, or PAI platform image.
	Select the instance type required to run the job. instance prices vary by specification. For billing details, see Billing for DLC.
	The number of compute nodes used in the DLC job. If you configure multiple nodes, each node is billed separately rather than sharing the same instance type. When you select nodes, consider the cost of each node and balance cost and performance.
	When is set to a purchased dedicated resource group, you can configure these parameters based on the purchased resource specifications.



	You can use advanced settings to improve training flexibility or meet specific training scenarios. When PyTorch is selected as the framework, you can configure advanced settings.
	The start command for each node. You need to configure hyperparameter variables `${<custom hyperparameter variable>}` in the command. Example: `python /mnt/data/examples/search/dlc_mnist/mnist.py --data_dir=/mnt/data/examples/search/data --save_model=/mnt/data/exmaples/search/model/model_${exp_id}_${trial_id} --batch_size=${batch_size} --lr=${lr} --metric_filepath=/mnt/data/examples/search/metric/metric_${exp_id}_${trial_id}` In this example, `${batch_size}` and `${lr}` are the defined hyperparameter variables.
Hyperparameter	Based on the hyperparameter variables configured in the start command, the hyperparameter list is automatically loaded. You need to configure the constraint type and search space for each hyperparameter: Constraint Type: the restriction applied to the hyperparameter. You can hover over the icon next to the constraint type to view the supported constraint types and descriptions. Search Space: the value range for the hyperparameter. The configuration method varies by constraint type. Click the icon and add values as prompted.

MaxCompute

Parameter

Description

Command

Configure this parameter to an SQL command or a PAI command for running Designer algorithm components. You need to configure hyperparameter variables ${<custom hyperparameter variable>} in the command. Example:

pai -name kmeans
    -project algo_public
    -DinputTableName=pai_kmeans_test_input
    -DselectedColNames=f0,f1
    -DappendColNames=f0,f1
    -DcenterCount=${centerCount}
    -Dloop=10
    -Daccuracy=0.01
    -DdistanceType=${distanceType}
    -DinitCenterMethod=random
    -Dseed=1
    -DmodelName=pai_kmeans_test_output_model_${exp_id}_${trial_id}
    -DidxTableName=pai_kmeans_test_output_idx_${exp_id}_${trial_id}
    -DclusterCountTableName=pai_kmeans_test_output_couter_${exp_id}_${trial_id}
    -DcenterTableName=pai_kmeans_test_output_center_${exp_id}_${trial_id};

In this example, ${centerCount} and ${distanceType} are the defined hyperparameter variables. For more configuration examples, see .

Hyperparameter

Based on the hyperparameter variables configured in the command, the hyperparameter list is automatically loaded. You need to configure the constraint type and search space for each hyperparameter:

Trial configuration

Parameter	Description
Metric Type	The type of metric used to evaluate a Trial. Valid values: summary: the final metric is extracted from TensorFlow summary files in OSS. table: the final metric is extracted from a MaxCompute table. stdout: the final metric is extracted from the standard output during job execution. json: the final metric is stored as a JSON file in OSS.
Method	During job execution, multiple intermediate metrics are generated. The method determines the final metric. Valid values: final: uses the last metric as the final metric for the Trial. best: uses the best metric obtained during job execution as the final metric for the Trial. avg: uses the average of all intermediate metrics obtained during job execution as the final metric for the Trial.
Metric Weight	When multiple metrics are considered, configure the metric name and weight. The system uses the weighted sum as the final evaluation metric for comparison. key: the metric name, which supports regular expressions. value: the corresponding weight. The weight can be negative, and the sum of weights does not need to be 1. Custom values are supported.
Metric Source	The source of the metric: When Metric Type is set to summary or json, configure a file path. Example: `oss://examplebucket/examples/search/pai/model/model_${exp_id}_${trial_id}`. When Metric Type is set to table, configure an SQL statement that returns the specific result. Example: `select GET_JSON_OBJECT(summary, '$.calinhara') as vrc from pai_ft_cluster_evaluation_out_${exp_id}_${trial_id}`. When Metric Type is set to stdout, configure a command keyword. Only `cmdx` or `cmdx;xxx, such as cmd1;worker` is supported.
optimization direction	The optimization direction for evaluating Trial results. Valid values: Larger is better Smaller is better
Model storage path	The path for storing the model. The path must contain `${exp_id}_${trial_id}` to distinguish models generated under different hyperparameter combinations. Example: `oss://examplebucket/examples/search/pai/model/model_${exp_id}_${trial_id}`.

Search configurations

Parameter	Description
Search Algorithm	An automated machine learning algorithm that finds better hyperparameter combinations for the next Trial based on the hyperparameter search space and the results and performance metrics of previous Trials. Supported algorithm types: TPE Random GridSearch Evolution GP PBT For more information about the algorithms, see Supported search algorithms.
Maximum Trials	The maximum number of Trials that the experiment can run.
Maximum Concurrent Trials	The maximum number of Trials that the experiment can run in parallel.
Enable EarlyStop	After you enable EarlyStop, if a Trial finds that the result is significantly poor when evaluating a specific hyperparameter combination, the evaluation process of the Trial can be terminated early without completing the full evaluation.
Start step	This parameter is available only when EarlyStop is enabled. It specifies the minimum number of evaluations that a Trial must complete before early stopping can be triggered.

What to do next

You can Experiment details at any time to monitor the experiment progress. By viewing the execution results of each Trial, you can obtain the optimal hyperparameter combination.
You can Manage experiments.

Appendix: Reference description

The following are configuration examples for MaxCompute job hyperparameter tuning:

Designer components used: K-means clustering and Clustering model evaluation.
The commands are configured as follows. cmd1 and cmd2 correspond to the commands of the two components, arranged in order. For the specific workflow, see MaxCompute k-means clustering.

cmd1

pai -name kmeans
    -project algo_public
    -DinputTableName=pai_kmeans_test_input
    -DselectedColNames=f0,f1
    -DappendColNames=f0,f1
    -DcenterCount=${centerCount}
    -Dloop=10
    -Daccuracy=0.01
    -DdistanceType=${distanceType}
    -DinitCenterMethod=random
    -Dseed=1
    -DmodelName=pai_kmeans_test_output_model_${exp_id}_${trial_id}
    -DidxTableName=pai_kmeans_test_output_idx_${exp_id}_${trial_id}
    -DclusterCountTableName=pai_kmeans_test_output_couter_${exp_id}_${trial_id}
    -DcenterTableName=pai_kmeans_test_output_center_${exp_id}_${trial_id};

cmd2

PAI -name cluster_evaluation
    -project algo_public
    -DinputTableName=pai_cluster_evaluation_test_input
    -DselectedColNames=f0,f1
    -DmodelName=pai_kmeans_test_output_model_${exp_id}_${trial_id}
    -DoutputTableName=pai_ft_cluster_evaluation_out_${exp_id}_${trial_id};