DLC FAQ

Ask Xiao PAI first

PAI Assistant (Xiao PAI) provides O&M and diagnostics for DSW instances, DLC jobs, and EAS services. Xiao PAI automatically diagnoses failures and recommends next steps.

Q: Job fails with "SupportsDistributedTraining false" error

• Cause: The training job is configured to use multiple instances (more than one node), but the model does not support distributed training.

• Solution: Set the number of nodes to 1.

Q: Job fails with "resource limiting triggered" error

This error occurs because the job exceeds the concurrent resource limit, which is currently set to a maximum of 2 GPUs. To resolve this, wait for other running jobs to finish, or submit a ticket to request a higher resource quota.

Q: What to do about an "exited with code 137" error?

If you encounter an "xxx exited with code 137" error, try using an instance with more memory, increasing the number of workers, or modifying your code to consume less memory.

In Linux, exit code 137 indicates the process was terminated by a SIGKILL signal, which usually points to an out-of-memory (OOM) error. You can check worker memory usage in the job details to confirm. To resolve the error, switch to an instance type with more memory, add more workers, or modify your code to consume less memory.

Q: Job fails or is dequeued

The status of a DLC job changes in the following sequence:

• What should I do if the job is stuck in the Environment Preparing status?

  If a job is stuck in the Environment Preparing state, a common cause is using a CPFS-type dataset for a distributed training job without specifying a VPC. To resolve this, you must recreate the job and configure both the CPFS-type dataset and a VPC. Ensure the selected VPC matches the one used by the CPFS file system. For more information, see Create a training job.

• What should I do if the job status is Failed?

  On the job details page, you can identify the failure cause by hovering over the icon next to the job status or by checking the instance operational logs. For more information, see View training job details.

Q: Switch resource type for an existing job?

No, you cannot change the resource type for a running job. You must create a new one. To simplify this, click Clone in the Actions column of the original job. This creates a new job with the same configuration, which you can then modify before submitting. For more information about billing, see Billing of DLC.

Q: Configure a multi-node, multi-GPU job

When you create a DLC job, you can configure the following startup command. For more configuration details, see Create a training job.

python -m torch.distributed.launch \ --nproc_per_node=2 \ --master_addr=${MASTER_ADDR} \ --master_port=${MASTER_PORT} \ --nnodes=${WORLD_SIZE} \ --node_rank=${RANK} \ train.py --epochs=100

Q: How to download a trained model locally?

To save your trained model, associate a dataset with your DLC job and configure the startup command to output results to the mounted directory. In the Environment Information section of the job configuration, add a custom dataset, set the Mount Path to /mnt/data/, and then add the command cp -r ./output /mnt/data to your Startup Command. This copies the model files produced during training to the mounted dataset directory.

After the job is complete, the model files are automatically saved to the mounted dataset directory. You can then access the file system corresponding to the mounted dataset and download the model files to your local machine.

• To learn how to associate a dataset when you submit a DLC job, see Create a training job.

• To learn how to download files from an OSS file system to your local machine, see Quick start.

• To learn how to download files from a NAS file system to your local machine, see Mount a file system in Function Compute.

Q: How do I use Docker images in DLC?

• Create a DLC job using a custom Docker image: Push your Docker image to Alibaba Cloud Container Registry (ACR) and add it as a custom image in your PAI workspace. When you create a DLC job, you can select this image to launch your instances.

  • To learn how to push a Docker image to ACR, see Push and pull images using an instance of a Personal Edition.

  • To learn how to add a PAI custom image, see Custom images.

• Install and use Docker within a DLC container: A DLC job runs inside a container. Therefore, you cannot install or use Docker within a DLC container.

Q: How to release nodes that finish early?

Problem description:

In a multi-worker distributed training job, data skew can cause some workers to finish their tasks earlier than others. By default, these completed workers continue to occupy their scheduled nodes until the entire job finishes.

Solution: You can configure the ReleaseResourcePolicy advanced parameter for PAI-DLC. By default, compute resources are released only after the entire job completes. If you set the parameter to pod-exit, resources are released as soon as a worker process exits.

Q: Job fails with "Stale file handle" error

Problem description:

Multiple workers fail with an "OSError: [Errno 116] Stale file handle" error when executing PyTorch's torch.compile (an AOT compilation feature). The failure occurs when the workers attempt to read a cache file.

Troubleshooting and analysis:

This error typically occurs in a Network File System (NFS) environment. It occurs when a client uses a file handle that has become stale, meaning the file was deleted or moved on the server.

Root cause:

This error stems from PyTorch's ahead-of-time (AOT) compilation, which caches computation graphs. The "stale file handle" error occurs because multiple workers concurrently read, write, or clean this cache, leading to race conditions where a worker tries to use a handle for a file that another process has already deleted.

Recommended solution:

• As a primary solution, force PyTorch to use a local tmpfs for caching by setting the following environment variable: TORCHINDUCTOR_CACHE_DIR=/dev/shm/torch_cache.

Alternative solutions:

1. Use Redis as a shared cache, as described in the official PyTorch documentation. This requires setting up a Redis service.

2. Review your CPFS NFS mount options. Using the noac option might mitigate the issue but can impact performance.

3. Disable caching entirely by setting TORCHINDUCTOR_CACHE_DIR="". This will negatively impact compilation performance.

Q: Dequeue fails with "roleMaximumResource" error

Problem description:

A customer's DLC has been queued for a long time, but resources are sufficient and no faults are detected on the nodes:

The job fails to dequeue, and the error message indicates a resource limit has been exceeded for the user's role. For example, the job requests 512 CPU cores, 4.00 TiB of memory, and 32 GPUs, but the quota for the PAI.WorkspaceOutsider role is much lower: {"CodeType":"ConfigRule","Code":"roleMaximumResource","Reason":"Current Limit is CPU: 184 limited by Role PAI.WorkspaceOutsider.Memory: 1800 limited by Role PAI.WorkspaceOutsider.GPU: 8 limited by Role PAI.WorkspaceOutsider."}.

Troubleshooting and analysis:

The initial suspicion might be a permissions issue with the RAM user in the PAI workspace. However, even after the user is granted administrator permissions, the job still fails, but the role in the error message changes to PAI.WorkspaceAdmin.

The detailed reason for the dequeue failure shows that the resource request still exceeds the quota for the new role: {"CodeType":"ConfigRule","Code":"roleMaximumResource","Reason":"Current Limit is CPU: 184 limited by Role PAI.WorkspaceAdmin. Memory: 1800 limited by Role PAI.WorkspaceAdmin. GPU: 8 limited by Role PAI.WorkspaceAdmin."}.

Root cause:

This is because the workspace administrator has also restricted the resource usage quota:

In the Resource Usage Quota settings of the PAI workspace, you can view the limits for different roles. These limits include Maximum Number of GPUs, Maximum Number of CPU Cores, and Maximum Memory. You can adjust these values by clicking Modify Configuration.

Solution:

To resolve this issue, ask the PAI workspace administrator to either increase the resource quota for your role to meet the job's requirements or remove the limit entirely.

Q: "Transport endpoint is not connected" error

Problem description:

After a DLC job starts, commands that access a mounted dataset path like /mnt/data, fail with a Transport endpoint is not connected error. This indicates that the NFS mount point is disconnected.

Common causes:

• VPC mismatch for CPFS: When using a CPFS dataset, the training job's VPC must match the CPFS file system's VPC. A mismatch will cause the mount to fail.

• Dataset not ready: The dataset created through AI Asset Management may still be in the process of initializing.

• Incorrect mount path: The mount path specified in the job configuration does not match the actual mount point of the dataset.

Solution:

1. In AI Asset Management, confirm that the dataset status is "Available".

2. If you are using a CPFS dataset, ensure the training job's VPC matches the CPFS file system's VPC. For more information, see the VPC configuration section in Create a training job.

3. Check the mount path: Run ls -la /mnt/data in the task logs to verify that the mount is accessible.

4. If you are using an OSS dataset, consider using the Storage Mount method instead of a dataset mount, as it can offer better compatibility.

Q: "Unable to attach or mount volumes" error

Problem description:

During startup, the DLC job gets stuck in the network initialization or environment preparation phase. The logs show the following error:

Unable to attach or mount volumes: unmounted volumes=[hostpath-volume], unattached volumes=[...]: timed out waiting for the condition

Common causes:

• Insufficient disk space on node: The local disk of the compute node is full, which prevents the hostpath-volume from being mounted.

• Storage backend failure: The storage service (such as NAS, CPFS, or OSS) has a temporary failure or connection timeout.

• Node resource contention: Multiple jobs are scheduled on the same node, causing the volume mount request to time out while waiting in a queue.

Solution:

1. Resubmit the job. This error is often transient, and creating a new job can resolve it.

2. Verify storage connectivity. Ensure that the NAS, CPFS, or OSS service associated with your dataset is healthy and that the VPC and security group configurations are correct.

3. If the problem persists, contact Alibaba Cloud technical support and provide the job ID and the complete error logs.