Experiment details

Updated at:

Monitor experiment details in real time, including basic information, trial list, run details for each trial, and logs.

Basic information

  1. Go to the AutoML page.

    1. Log on to the PAI console.

    2. In the left-side navigation pane, click Workspaces. On the Workspaces page, click the name of the workspace that you want to manage.

    3. In the left-side navigation pane, choose Model Training > AutoML.

  2. On the experiment list page, click an experiment name to go to the Experiment Details page.

  3. On the Basic Information tab of the Experiment Details page, you can view the basic configuration, trial configuration, execution configuration, search configuration, and trial execution progress.

    • In the Basic Settings section, you can view the experiment ID, status, and other details. An experiment can have one of the following statuses:

      • CREATED: The system is initializing the experiment, which includes creating management processes, parsing configuration modules, and submitting tasks to backend services such as Deep Learning Containers (DLC) or MaxCompute. No trials are running at this stage.

      • FINISHED: All trials have completed and at least one trial is FINISHED.

      • FAILED: All trials have failed, or you manually stopped all trials.

      • RUNNING: At least one trial is running.

      • TERMINATING: A temporary state after you manually stop the experiment.

      • USER_CANCELED: You manually stopped the experiment and all processes have terminated.

      • NO_MORE_TRIAL: The experiment has reached the maximum number of configured trials and will not generate new ones. After the last running trial completes, the status changes to FINISHED.

      • TUNER_NO_MORE_TRIAL tuner: The tuner has stopped generating new trials because it lacks sufficient information to determine the next hyperparameter combination.

    • In the Trial Configuration, Execution Configuration, and Search Configuration sections, you can view the experiment's parameter configurations.

    • In the Trial Execution Progress section, you can view the trials' execution progress and status statistics.

Trial list

  1. Go to the Experiment Details page. For more information, see View basic information.

  2. The Trial List tab displays a list of all trials automatically generated for the experiment.

    The experiment uses the configured algorithm to generate hyperparameter combinations, creating one trial per combination. Each trial corresponds to a Deep Learning Containers (DLC) task or one or more MaxCompute tasks, depending on the execution configuration.

Trial run details

The Trial List tab shows the details of each trial, including its status, final metric, and hyperparameter combination.

  • Status: A trial can be in one of the following states:

    • RUNNING: The trial is in progress.

    • FINISHED: The trial completed successfully.

    • FAILED: The trial failed to complete.

    • USER_CANCELED: You manually stopped the trial.

    • EARLY_STOPPED: The trial was stopped early.

  • Final Metric: The system calculates a weighted sum of the configured metric weights as the final evaluation metric.

  • Hyperparameter Combination: The configured algorithm generates a unique hyperparameter combination for each trial.

Compare the Final Metric of each trial against the configured Optimization Direction to identify the optimal hyperparameter combination.

Logs

If an experiment or a trial fails, use logs to troubleshoot the failure.

Experiment logs

  1. Go to the Experiment Details page. For more information, see View basic information.

  2. Click the Log button on the right side of the page. Alternatively, on the Basic Information tab, in the Basic Settings section, click the Log button next to Status.

  3. In the Experiment Log panel, review the log details. Two types of logs are available:

    • nnimanager.log

      Records experiment management details, such as start and stop events, and error messages.

    • dispatcher.log

      Records task scheduling and resource management details, such as trial assignment and resource allocation.

Trial logs

  1. Go to the Experiment Details page. For more information, see View basic information.

  2. On the Trial List tab, find the target trial and click Log in the Actions column.

  3. In the Trial Logs panel, review the trial's log details.

    A trial generates three types of logs:

    • trial.log

      Records the detailed execution process of the trial.

    • stdout

      Contains the standard output of the trial, including print statements and other outputs, to help you understand debugging information and execution results.

    • stderr

      Contains the standard error output of the trial, including exceptions, error messages, and warnings. When a trial encounters an error, this log provides detailed information for troubleshooting.