DocsICP FilingConsole
官方文档
首页

PS-SMART binary classification on MaxCompute

更新时间：
复制 MD 格式
产品详情
我的收藏

This topic explains how to submit a hyperparameter tuning experiment on MaxCompute. The experiment runs the PS-SMART Binary Classification, Prediction, and Binary Classification Evaluation components to find an optimal hyperparameter combination for the PS-SMART algorithm.

Prerequisites

Step 1: Prepare data

This example uses a bank customer product subscription prediction dataset that has undergone feature engineering. Follow these steps to prepare the training and test datasets:

  1. Run the following SQL commands in the MaxCompute client to create the bank_train_data and bank_test_data tables. For details on installing and configuring the MaxCompute client, see Connect by using the local client (odpscmd).

    create table bank_train_data(
id bigint ,age double ,job double ,marital double ,education double ,default double ,housing double ,loan double ,contact double ,
month double ,day_of_week double ,duration double ,campaign double ,pdays double ,previous double ,poutcome double ,emp_var_rate double ,
cons_price_index double ,cons_conf_index double ,lending_rate3m double ,nr_employed double ,subscribe bigint
);
create table bank_test_data(
id bigint ,age double ,job double ,marital double ,education double ,default double ,housing double ,loan double ,contact double ,
month double ,day_of_week double ,duration double ,campaign double ,pdays double ,previous double ,poutcome double ,emp_var_rate double ,
cons_price_index double ,cons_conf_index double ,lending_rate3m double ,nr_employed double ,subscribe bigint
);

  2. Run the Tunnel command in the MaxCompute client to upload the training dataset and test dataset to the bank_train_data and bank_test_data tables, respectively. For more information about Tunnel commands, see Tunnel commands.

    -- Upload the training dataset to the bank_train_data table. Replace xx/train_data.csv with the actual path of train_data.csv.
tunnel upload  xx/train_data.csv bank_train_data;
-- Upload the test dataset to the bank_test_data table. Replace xx/test_data.csv with the actual path of test_data.csv.
tunnel upload  xx/test_data.csv bank_test_data;

Step 2: Create an experiment

On the Create Experiment page, configure the key parameters as follows. For more information about other parameters, see Create an experiment. After you configure the parameters, click Submit.

  1. Configure execution settings.

    Parameter

    Description

    Job type

    Select MaxCompute.

    Command

    Configure the following five commands.

    • cmd1: Run the PS-SMART Binary Classification component on the training data to build a binary classification model. For more information about the parameters, see PS-SMART Binary Classification.

      PAI -name ps_smart
    -project algo_public
    -DinputTableName='bank_train_data'
    -DmodelName='bi_ps_${exp_id}_${trial_id}'
    -DoutputTableName='bi_model_output_${exp_id}_${trial_id}'
    -DoutputImportanceTableName='bi_imp_${exp_id}_${trial_id}'
    -DlabelColName='subscribe'
    -DfeatureColNames='age,job,marital,education,default,housing,loan,contact,month,day_of_week,duration,campaign,pdays,previous,poutcome,emp_var_rate,cons_price_index,cons_conf_index,lending_rate3m,nr_employed'
    -DenableSparse='false'
    -Dobjective='binary:logistic'
    -Dmetric='error'
    -DfeatureImportanceType='gain'
    -DtreeCount='${tree_count}'
    -DmaxDepth='${max_depth}'
    -Dshrinkage="0.3"
    -Dl2="1.0"
    -Dl1="0"
    -Dlifecycle="3"
    -DsketchEps="0.03"
    -DsampleRatio="1.0"
    -DfeatureRatio="1.0"
    -DbaseScore="0.5"
    -DminSplitLoss="0"

    • cmd2: Drop the prediction result table.

      drop table if exists bi_output_${exp_id}_${trial_id};

    • cmd3: Run the Prediction component to make predictions on the input data by using the classification model generated in cmd1. For more information about the parameters, see Prediction.

      PAI -name prediction
    -project algo_public
    -DinputTableName='bank_test_data'
    -DmodelName='bi_ps_${exp_id}_${trial_id}'
    -DoutputTableName='bi_output_${exp_id}_${trial_id}'
    -DfeatureColNames='age,job,marital,education,default,housing,loan,contact,month,day_of_week,duration,campaign,pdays,previous,poutcome,emp_var_rate,cons_price_index,cons_conf_index,lending_rate3m,nr_employed'
    -DappendColNames='subscribe,age,job,marital,education,default,housing,loan,contact,month,day_of_week,duration,campaign,pdays,previous,poutcome,emp_var_rate,cons_price_index,cons_conf_index,lending_rate3m,nr_employed'
    -DenableSparse='false'
    -Dlifecycle='3';

    • cmd4: Run the Binary Classification Evaluation component to evaluate the model performance based on the prediction results from cmd3. For more information about the parameters, see Binary Classification Evaluation.

      PAI -name evaluate -project algo_public
    -DoutputDetailTableName='bi_0804_${exp_id}_${trial_id}_outputDetailTable'
    -DoutputMetricTableName='bi_0804_${exp_id}_${trial_id}_outputMetricTable'
    -DlabelColName='subscribe'
    -DscoreColName='prediction_score'
    -DpositiveLabel='1'
    -DbinCount='1000'
    -DdetailColName='prediction_detail'
    -DlabelMatch='true'
    -DinputTableName='bi_output_${exp_id}_${trial_id}';

    • cmd5: Extract evaluation metrics from the evaluation result table of cmd4.

      INSERT OVERWRITE TABLE ps_smart_classification_metrics PARTITION(pt='${exp_id}_${trial_id}')
    SELECT /*+MAPJOIN(b,c,d)*/ REGEXP_EXTRACT(a.data_range, '\\\((.*?),') as threshold,
    a.recall, a.precision, a.f1_score, c.value as auc, d.value as ks
    FROM (SELECT recall, precision, f1_score, data_range, 'AUC' auc, 'KS' ks from bi_0804_${exp_id}_${trial_id}_outputDetailTable) a
    JOIN bi_0804_${exp_id}_${trial_id}_outputMetricTable b
    on b.name='F1 Score' AND a.f1_score=b.value
    JOIN bi_0804_${exp_id}_${trial_id}_outputMetricTable c
    ON c.name=a.auc
    JOIN bi_0804_${exp_id}_${trial_id}_outputMetricTable d
    ON d.name=a.ks;

    Hyperparameter

    Configure the constraint type and search space for each hyperparameter as follows:

    • tree_count:

      • Constraint Type: choice.

      • Search Space: Click image.png and add three enumerated values: 50, 100, and 150.

    • max_depth:

      • Constraint Type: choice.

      • Search Space: Click image.png and add three enumerated values: 6, 8, and 10.

    These settings generate nine hyperparameter combinations. The experiment creates a Trial for each combination. In each Trial, the experiment runs the PS-SMART Binary Classification and Binary Classification Evaluation components with one set of hyperparameters.

  2. Configure Trial settings.

    Parameter

    Description

    Metric Type

    Select table.

    Method

    Select best.

    Metric Weight

    Configure the following metric weights:

    • key: recall, value: 0.5

    • key: precision, value: 0.25

    • key: auc, value: 0.25

    Metric Source

    Set to select * from ps_smart_classification_metrics where pt='${exp_id}_${trial_id}';.

    Optimization

    Select Maximize.

    Model name

    Set to bi_ps_${exp_id}_${trial_id}.

  3. Configure search settings.

    Parameter

    Description

    Search Algorithm

    Select TPE.

    Maximum Trials

    Set to 5.

    Maximum Concurrent Trials

    Set to 2.

Step 3: View experiment details and results

  1. On the Experiments page, click the experiment name to go to the Experiment Details page. The Basic information tab shows that the experiment status is FINISHED, and the Trial summary indicates 5 completed Trials, with 0 failed and 0 running. The Trial configuration shows the Metric type is table, the Method is best, and the Metric weight is precision:0.25, recall:0.5, and auc:0.25. The metric source is a SQL statement, and the model path is bi_ps_${exp_id}_${trial_id}. The search configuration confirms the Search algorithm is TPE, Maximum trials is 5, Maximum concurrent trials is 2, the Optimization direction is maximize, early stopping is enabled, and the start step is 5. On this page, you can view the progress and status of the Trials. The experiment automatically creates five Trials based on the configured search algorithm and maximum number of trials.

  2. Click the Trials tab to view a list of all Trials automatically generated for the experiment, along with each Trial's status, final metric, and hyperparameter combination. On the Trial list tab, a table displays information for each Trial, including its status, final metric, current metrics (such as auc, default, precision, and recall), hyperparameters (tree_count and max_depth), creation time, and duration. All Trials are complete with a status of FINISHED. The final metric (auc) ranges from 0.66066 to 0.68894. Because the configured optimization direction is Maximize, the experiment identifies the hyperparameter combination that resulted in the highest final metric of 0.68894 (tree_count: 50, max_depth: 8) as the best-performing one.

Previous：MaxCompute k-means clusteringNext：Best practices for DLC MNIST training