DLC FAQ

Ask Xiao PAI first

The PAI AI assistant (Xiao PAI) provides detailed guidance and step-by-step instructions for all PAI products. It can diagnose DSW instances, DLC jobs, and EAS services—automatically identifying failure causes and recommending next steps.

Q: A DLC job has been queued for a long time, but resource monitoring shows sufficient overall capacity. How do I troubleshoot this?

The resource monitoring page shows the total quota for the resource pool, but the scheduler allocates resources based on what's available on individual nodes. Even when the cluster has enough total resources, a job remains queued if no single node can satisfy the full resource combination the job requires (such as CPU + GPU together).

Check the following common causes:

Q: Model training fails with "SupportsDistributedTraining false, please set InstanceCount=1"

• Cause: The training job is configured to use multiple instances (node count greater than 1), but the model doesn't support distributed training.

• Solution: Set the node count to 1.

Q: Model training fails with "failed to compose dlc job specs, resource limiting triggered, you are trying to use more GPU resources than the threshold"

Training jobs are limited to a maximum of two GPU units running concurrently. If this threshold is exceeded, resource limiting is triggered. Wait for running training jobs to finish before starting new ones, or submit a ticket to request a quota increase.

Q: A job exits with error code 137. What should I do?

Error code 137 in Linux means the process was forcibly terminated by a SIGKILL signal. The most common cause is excessive memory usage, also known as an Out of Memory (OOM) error.

When you encounter this error (for example, "xxx exited with code 137"), check the memory utilization shown in the worker details of the job. Based on the findings, try one or more of the following:

• Switch to an instance type with more memory.

• Increase the number of workers.

• Reduce the memory allocation in your code.

Q: What should I do when a DLC job status shows "Failed" or "Dequeued"?

DLC job execution follows this status sequence:

• What to do when job status is "Preparing environment"

  If a job stays in Preparing environment for an extended period, the likely cause is that the distributed training job uses a CPFS dataset but no VPC is configured. Recreate the job with CPFS dataset configured, add a VPC, and make sure the VPC you select matches the one used by CPFS. For details, see Create a training job.

• What to do when job status is "Failed"

  On the job details page, hover over the icon next to the job status, or check the instance operation log to identify the initial cause of failure. For details, see Training details.

Q: Can I switch a DLC job from public resources to dedicated resources?

To switch resource types, you need to create a new job. In the Actions column of the original job, click Clone to create a new job that reuses the original configuration, so you don't have to re-enter all the parameters. For billing details, see Billing for DLC.

Q: How do I configure multi-node, multi-GPU training in DLC?

To enable multi-node, multi-GPU training, add the following launch command when creating a DLC job. For full configuration details, see Create a training job.

python -m torch.distributed.launch \ --nproc_per_node=2 \ --master_addr=${MASTER_ADDR} \ --master_port=${MASTER_PORT} \ --nnodes=${WORLD_SIZE} \ --node_rank=${RANK} \ train.py --epochs=100

Q: How do I download a trained model from PAI-DLC to my local machine?

When submitting a DLC job, attach a dataset and configure the launch command to write training output to the mounted dataset directory. In the Environment settings of the job, add a custom dataset and set the Mount path to /mnt/data/. Then, in the Launch command, run cp -r ./output /mnt/data to copy the model files to the mounted dataset directory.

After training completes, the model files are saved in the mounted dataset directory. Access the file system associated with the dataset to download the files to your local machine.

• For how to attach a dataset when submitting a DLC job, see Create a training job.

• For how to download files from an OSS file system to your local machine, see Console quick start.

• For how to download files from a NAS file system to your local machine, see Mount a file system on a Function Compute function.

Q: How do I use a Docker image in DLC?

• Use a Docker image to create a DLC job: Push the Docker image to Alibaba Cloud Container Registry (ACR), then add it to your PAI workspace as a custom image. You can then select that image when creating a DLC job.

  • For how to push a Docker image to ACR, see Push and pull images in a Personal Edition instance.

  • For how to add a custom image in PAI, see Custom images.

• Install and use Docker inside a DLC container: DLC jobs run inside containers, so installing or running Docker within a DLC container isn't supported.

Q: Some nodes in a DLC job finish early. How do I release their resources for other jobs?

Background:

In a multi-worker distributed DLC job, workers may finish at different times—for example, due to data skew. By default, completed workers continue to hold their allocated nodes until the entire job finishes.

Solution: Configure the ReleaseResourcePolicy advanced parameter. By default, resources are held until the entire job completes. Set this parameter to pod-exit to release each worker's resources as soon as that worker finishes.

Q: A DLC job fails with "OSError: [Errno 116] Stale file handle"

Background:

Multiple workers encounter this error while running PyTorch's torch.compile (AOT compilation). The error occurs when a process holds a file handle that becomes invalid because the file was deleted or moved on the server side.

Root cause:

PyTorch's AOT compilation caches optimized computation graphs in the file system (default: /tmp, typically local tmpfs). In cluster environments, this cache may land on CPFS (a distributed file system mounted over NFS). NFS is sensitive to file deletion, and the following conditions trigger a stale file handle:

• PyTorch automatically cleans up old cache entries, or another process deletes the cache directory.

• The NFS server moves, deletes, or changes permissions on a file, but the client still holds the old handle.

• NFS clients cache file attributes by default, so clients may not detect server-side changes.

These conditions are more likely in multi-worker jobs: CPFS is NFS-based and prone to stale handles under high-concurrency access, especially for short-lived cache files. Concurrent workers reading, writing, or cleaning the cache simultaneously can cause race conditions.

Primary solution:

• Force the cache to use local tmpfs by setting the environment variable TORCHINDUCTOR_CACHE_DIR=/dev/shm/torch_cache.

Alternative solutions:

1. Use Redis as a shared cache backend following the official PyTorch documentation (requires a Redis service).

2. Adjust the NFS mount options for CPFS (the noac option may reduce stale handle occurrences but can affect performance).

3. Disable the cache by setting TORCHINDUCTOR_CACHE_DIR="". This eliminates the error but also removes compilation performance benefits.

Q: A DLC job fails to dequeue with error code "roleMaximumResource"

Background:

A DLC job stays in the queue even though resources appear available and no node failures are detected.

The job requests CPU: 512, memory: 4.00 TiB, and GPU: 32, which exceeds the quota for the PAI.WorkspaceOutsider role (CPU: 184, memory: 1800, GPU: 8). The full error is: {"CodeType":"ConfigRule","Code":"roleMaximumResource","Reason":"Current Limit is CPU: 184 limited by Role PAI.WorkspaceOutsider.Memory: 1800 limited by Role PAI.WorkspaceOutsider.GPU: 8 limited by Role PAI.WorkspaceOutsider."}.

Root cause:

The workspace administrator has configured per-role resource usage limits. After the primary account granted administrator permissions, the error message changed to reference the PAI.WorkspaceAdmin role:

{"CodeType":"ConfigRule","Code":"roleMaximumResource","Reason":"Current Limit is CPU: 184 limited by Role PAI.WorkspaceAdmin. Memory: 1800 limited by Role PAI.WorkspaceAdmin. GPU: 8 limited by Role PAI.WorkspaceAdmin."}

This confirms that resource usage limits are enforced at the workspace level. In the workspace Resource Usage Limits settings, the maximum GPU cards, CPU cores, and memory allowed per role (such as administrator, algorithm developer, or algorithm operator) are listed. Click Modify to update these limits.

Solution:

Ask the workspace administrator to remove the resource limit or raise it to at least the amount the DLC job requires.

Q: Accessing /mnt/data fails with "Transport endpoint is not connected"

Background:

After a DLC job starts, accessing the mounted dataset path (such as /mnt/data) in the launch command fails with: Transport endpoint is not connected. This error indicates the NFS mount point has disconnected.

Common causes:

• CPFS VPC mismatch: When using a CPFS dataset, the VPC of the training job must match the VPC of the CPFS file system. A mismatch causes the mount to fail.

• Dataset not ready: A dataset created through AI asset management may still be initializing.

• Incorrect mount path: The mount path configured in the job doesn't match the actual mount point of the dataset.

Solution:

1. In AI asset management, confirm the dataset status is "Available."

2. If you're using a CPFS dataset: when creating the training job, make sure the VPC you configure matches the CPFS file system's VPC. For VPC configuration details, see Create a training job.

3. Check the mount path: run ls -la /mnt/data in the job log to verify the mount is accessible.

4. If you're using an OSS dataset: use storage mounting instead of dataset mounting for better compatibility.

Q: A DLC job is stuck in "Preparing environment" with error "Unable to attach or mount volumes"

Background:

During startup, the DLC job gets stuck in the network initialization or environment preparation phase. The log shows the following error:

Unable to attach or mount volumes: unmounted volumes=[hostpath-volume], unattached volumes=[...]: timed out waiting for the condition

Common causes:

• Insufficient node disk space: The local disk on the compute node is full, preventing the hostpath-volume from being mounted.

• Storage backend issues: NAS, CPFS, or OSS storage is experiencing a temporary failure or connection timeout.

• Node resource contention: Multiple jobs are scheduled to the same node simultaneously, causing volume mount operations to time out while waiting in queue.

Solution:

1. Resubmit the job: In most cases this is a transient issue and resubmitting the job resolves it.

2. Check storage connectivity: Confirm the NAS, CPFS, or OSS service associated with the dataset is healthy and that the VPC and security group configurations are correct.

3. If the issue persists, contact Alibaba Cloud technical support and include the job ID and full error log.