Model Gallery FAQ

Ask Xiao PAI first

PAI Assistant (Xiao PAI) provides O&M and diagnostics for DSW instances, DLC jobs, and EAS services. Xiao PAI automatically diagnoses failures and recommends next steps.

Q: How do I check whether deep thinking is enabled for a model evaluation?

Check the default deployment configuration on the model description page. All models are evaluated using their default deployment settings.

Q: How do I deploy a model after training?

1. Go to PAI-Model Gallery > Task Management.

2. Find the completed training task and click it to open the details page.

3. Click Deploy in the top-right corner.

Q: How do I troubleshoot a failed training task?

When a training task fails, check the task diagnostics or logs. A common cause is incorrectly formatted dataset.

Two ways to find the error details:

• Task diagnostics: In PAI-Model Gallery, go to Task Management > Training Tasks. Click the failed task to open its Task Details page. Hover over the Failed status to view an error summary.

  

• Task logs: On Task Management > Training Tasks, select the Task log tab to view the full error message.

  

  Common error messages and solutions:

  Error type	Error message	Solution
  Input/output errors	ValueError: output channel ${your OSS uri} must be directory	Verify that the output path is a directory and not a file.
  	ValueError: train must be a file	Verify that the input path points to a file and not a directory.
  	FileNotFoundError	Ensure that a valid file exists at the specified input path.
  	JSONDecodeError	Verify that the JSON file is formatted correctly.
  	ValueError: Input data must be a json file or a jsonl file!	Ensure that the file is in JSON or JSONL format.
  	KeyError: ${some key name}	Review the model documentation and verify that all key-value pairs in your dataset match the required format. This error often occurs with JSON datasets.
  	ValueError: Unrecognized model in /ml/input/data/model/.	PyTorch does not recognize the model file format.
  	UnicodeDecoderError	Verify that the file uses the correct character encoding.
  	Input/output error	Ensure that you have read permission for the input path and write permission for the output path.
  	NotADirectoryError: [Errno 20] Not a directory:	Verify that the specified path is a directory.
  Hyperparameter settings	ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 0 (pid: 51) of binary: /usr/bin/python (and there are no related subprocess logs)	Insufficient RAM causes an out-of-memory (OOM) error when loading the model. Select an instance type with more memory.
  	torch.cuda.OutOfMemoryError: CUDA out of memory	Insufficient GPU memory. Choose a GPU instance with more VRAM, or reduce memory-intensive hyperparameters like lora_dim or batch_size.
  	ValueError: No closing quotation	system prompt or another string parameter contains an unclosed quotation mark ("), preventing training command generation. Ensure all quotation marks are properly paired.
  Resource configuration of an instance type	Exception: Current loss scale already at minimum - cannot decrease scale anymore. Exiting run	Precision issue. The model uses BF16 format, which requires a GPU with Ampere or newer architecture (e.g., A10, A100). Training on older GPUs converts parameters to FP16, which can cause this underflow error.
  	RuntimeError: CUDA error: uncorrectable ECC error encountered	Hardware fault on the GPU instance. Retry the training task. If it fails again, try a different instance type or region.
  	MemoryError: WARNING Insufficient free disk space	Disk is full. Select an instance type with more disk space.
  User limit errors	failed to compose dlc job specs, resource limiting triggered, you are trying to use more GPU resources than the threshold	GPU quota exceeded. Training tasks are limited to 2 concurrent GPUs by default. Wait for a running task to complete, or submit a ticket to request a quota increase.

Q: How do I perform online debugging for a deployed model?

After deploying your model as a service on PAI-EAS, use the built-in online debugging tool.

1. Find the deployed service: Go to PAI-Model Gallery > Task Management > Deployment Tasks. You are redirected to the PAI-EAS console where you can view the deployed service.

   

2. Open the debugging tool: On the PAI-EAS page, locate your service. In the Operation column, click Online Debugging.

   

3. Configure and send a request:

   1. Find the required request format on the model description page in PAI-Model Gallery. For example, a model deployed using BladeLLM might expect a POST request to /v1/chat/completions with a specific JSON body.

      

      

   2. In the Online Debugging UI, append the path (e.g., /v1/chat/completions) to the Request URL.

      

   3. Populate the Request Body using the example from the model's description page and click Send Request.

Q: Why am I getting a "503 no healthy upstream" error when calling my deployed model?

A 503 Service Unavailable error with message no healthy upstream indicates the service instance is out of resources and cannot process new requests.

• Cause: Instance resources (CPU, memory, or GPU memory) are fully utilized, leaving no capacity for new requests.

• Solution: The correct action depends on your resource type:

  • Public Resources: The shared resource pool is temporarily overloaded. Try again during off-peak hours, or redeploy to a different region or instance specification.

  • Dedicated Resources (EAS Resource Group): Your resource group is underscaled. Ensure the group has sufficient CPU, memory, and GPU memory. Maintain at least 20% resource buffer.

Q: Why do I get "SupportsDistributedTraining false, please set InstanceCount=1" error during training?

• Cause: The selected model does not support distributed training, but the task was configured to run on multiple nodes (InstanceCount > 1).

• Solution: Set Number of Nodes to 1 in your training task configuration and restart the task.