Training details

Basic information and configuration

1. Log in to the PAI console. From the top navigation bar, select the target region and, on the right, the target workspace. Then, click Enter DLC.

2. Click the name of the target job to open its overview page.

3. The Overview tab displays the job's basic information, environment information, and resource information.

   The top of the page displays the job's status (for example, Succeeded), duration, compute type (for example, General-purpose computing), job type (for example, PyTorchJob), and resource group, along with a timeline of its phases from creation to completion. The basic information includes the job name, job ID, and tags. The resource information includes the resource type, number of workers, and instance type (for example, ecs.gn7i-c8g1.2xlarge). The environment information includes the node image URL, dataset mount configuration, and execution command. The page also contains configuration sections such as Fault Tolerance and Diagnosis (with toggles for Auto Fault Tolerance and Health Check), Network Information (with details on the Virtual Private Cloud (VPC), security group, and vSwitch), and Roles and Permissions (with details on the Instance RAM role and visibility).

4. Click the job name at the top of the page to open a drop-down list of jobs. The list's fuzzy search by name or ID allows you to quickly switch between running and historical instances.

Job events

Event logs record the progress of job scheduling and resource allocation. You can use these events to identify and troubleshoot issues.

• Switch to the Event tab to view job event logs.

  The Event tab displays an event timeline on the left, showing stages such as job creation, environment preparation, job execution, and job completion, along with their corresponding time ranges. On the right, the event log panel shows details of the PyTorchJob lifecycle events, such as Job Queued, Dequeued, Pod Created, Service Created, Scheduled, Running, and Succeeded.

• On the Overview tab, go to the Instance section. In the Actions column for an instance, click Log, and then select the System Log tab to view node-specific event logs.

  The system log records the complete lifecycle events of a pod, including status changes (ResourcePurchasing → NetworkInitializing → Initializing → ImagePulling → WaitingForRun → Running → Succeeded), image pull records, and container creation and startup events.

Resource metrics

Key metrics include GPU utilization, GPU memory usage, CPU utilization, memory usage, and Network I/O. By monitoring these resource metrics in real time, you can understand the resource demands of your job, track utilization and consumption, and plan for resource optimization.

Switch to the Monitoring tab to view the job's resource metrics.

If you create a training job with a resource quota, you can also use the following monitoring features:

1. Metrics are available by Job Dimension, Pod Dimension, and GPU Dimension.

   The top of the Monitoring page displays the job lifecycle timeline, showing stages such as Job Created, Queued, Preparing Environment, Running, and Succeeded, along with the time taken for each stage. Below the timeline, you can switch between the Job Dimension, Pod Dimension, and GPU Dimension tabs to view the corresponding metrics. The GPU Dimension tab shows line charts for GPU utilization and GPU memory usage.

2. You can use interactive features, including filtering by time and metric type, and categorizing metrics. Click More to customize the metric view and sort the metrics. The DLC monitoring view allows you to create a personalized dashboard by selecting and reordering metrics. You can select key performance indicators based on your business needs and adjust their display priority by dragging and dropping them to focus on what matters most.

   Available GPU monitoring metrics include GPU utilization, Total GPU Memory, GPU memory usage, GPU Memory Used, GPU Memory Interface Utilization, GPU Memory Bandwidth Usage, GPU SM Core Utilization, GPU Power Consumption, and GPU Temperature. After selecting the desired metrics, drag them in the Sort metrics area on the right to adjust their display order, and then click OK to save your configuration.

3. DLC also supports monitoring and alerting to track the resource levels of your job in real time. For more information, see Training Monitoring and Alerting.

Job logs

If a job runs abnormally or you need to review its execution history, you can view the job logs in one of the following ways:

• On the Overview tab, go to the Instance section. In the Actions column for an instance, click Log to view the output log of that specific node.

• Switch to the Log tab to search for log events by keyword. For more information, see Query Aggregated Logs by Keyword. Here are some quick tips for the query syntax:

  - Basic query: error: finds logs containing "error".
  - Multi-word query: "Unexpected result": finds logs containing both "Unexpected" and "result".
  - Wildcard query: error*: finds words starting with "error". Special characters are not supported.
  - Phrase query: #"abc$def": matches the exact phrase "abc$def".
  Delimiters: Logs are split by delimiters. Delimiters in keywords are treated as null characters. To search for content containing delimiters, use a phrase query. Common delimiters include: \n\t\r,;[]{}()&^*#@~=<>/\?:'"

  On the Log page, the left side displays the instance list, and the right side displays the user logs for the selected instance. In the example, the training log shows validation results from Epoch 16 to Epoch 18, and the final output is Model saved with accuracy: 98.96%, indicating that the model training has completed successfully.

Audit logs

PAI is integrated with ActionTrail. You can view and search the last 90 days of DLC action events for your Alibaba Cloud account in ActionTrail. For more information, see ActionTrail.

Restart records

If you enabled Auto Fault Tolerance or Health Check (blocklist and rerun) when creating the job, you can click the Restart Count to go to the restart records page. This page displays information about restarts, including the restart count, restart time, restart reason, restart result, and restart duration.

• In the list of restart records, click Error Details to view detailed information about a specific restart, including the restart count, restart time, node name, instance name, error code, error message, and error source.

• Click View Aggregated Error Details to display the complete list of restart records with their details.

Related documentation

You can manage jobs based on their status. For more information, see Manage Training Jobs.