inference-nv-pytorch 25.11-Container Compute Service(ACS)-阿里云帮助中心

Key features and bug fixes

Key features

This release provides images for two CUDA versions: CUDA 12.8 and CUDA 13.0.
- The CUDA 12.8 image supports only the amd64 architecture.
- The CUDA 13.0 image supports both amd64 and aarch64 architectures.
Upgraded PyTorch to 2.9.0.
Upgraded deepgpu-comfyui to 1.3.2 and the deepgpu-torch optimization component to 0.1.12+torch2.9.0cu128 in the CUDA 12.8 image.
Upgraded vLLM to v0.11.2 in the CUDA 13.0 amd64 image, and SGLang to v0.5.5.post3 in the CUDA 12.8 and 13.0 SGLang-based images.

Bug fixes

None

Image name	inference-nv-pytorch
Image tag	25.11-vllm0.11.1-pytorch2.9-cu128-20251120-serverless	25.11-sglang0.5.5.post3-pytorch2.9-cu128-20251121-serverless	25.11-vllm0.11.1-pytorch2.9-cu130-20251120-serverless		25.11-sglang0.5.5.post3-pytorch2.9-cu130-20251121-serverless
Supported architecture	amd64	amd64	amd64	aarch64	amd64	aarch64
Use case	Large model inference	Large model inference	Large model inference	Large model inference	Large model inference	Large model inference
Framework	pytorch	pytorch	pytorch	pytorch	pytorch	pytorch
Requirements	NVIDIA driver release >= 570	NVIDIA driver release >= 570	NVIDIA driver release >= 580	NVIDIA driver release >= 580	NVIDIA driver release >= 580	NVIDIA driver release >= 580
System components	Ubuntu 24.04 Python 3.12 Torch 2.9.0+cu128 CUDA 12.8 diffusers 0.35.2 deepgpu-comfyui 1.3.2 deepgpu-torch 0.1.12+torch2.9.0cu128 flash_attn 2.8.3 flashinfer-python 0.5.2 imageio 2.37.2 imageio-ffmpeg 0.6.0 ray 2.51.1 transformers 4.57.1 triton 3.4.0 torchaudio 2.8.0+cu128 torchvision 0.24.0+cu128 vllm 0.11.1 xfuser 0.4.5 xgrammar 0.1.25 ljperf 0.1.0+477686c5	Ubuntu 24.04 Python 3.12 Torch 2.9.0+cu128 CUDA 12.8 diffusers 0.35.2 decord 0.6.0 decord2 2.0.0 deepgpu-comfyui 1.3.2 deepgpu-torch 0.1.12+torch2.9.0cu128 flash_attn 2.8.3 flash_mla 1.0.0+1408756 flashinfer-python 0.5.2 imageio 2.37.2 imageio-ffmpeg 0.6.0 ray 2.51.1 transformers 4.57.1 sgl-kernel 0.3.17.post1 sglang 0.5.5.post3 xgrammar 0.1.25 triton 3.5.0 torchao 0.9.0 torchaudio 2.8.0+cu128 torchvision 0.24.0+cu128 xfuser 0.4.5 ljperf 0.1.0+477686c5	Ubuntu 24.04 Python 3.12 Torch 2.9.0+cu130 CUDA 13.0.2 diffusers 0.35.2 flash_attn 2.8.3 flashinfer-python 0.5.2 imageio 2.37.2 imageio-ffmpeg 0.6.0 ray 2.51.1 transformers 4.57.1 triton 3.5.0 torchaudio 2.9.0+cu130 torchvision 0.24.0+cu130 vllm 0.11.2 xfuser 0.4.5 xgrammar 0.1.25 ljperf 0.1.0+d0e4a408	Ubuntu 24.04 Python 3.12 Torch 2.9.0+cu130 CUDA 13.0.2 diffusers 0.35.2 flash_attn 2.8.3 flashinfer-python 0.5.2 transformers 4.57.1 ray 2.51.1 vllm 0.11.1 triton 3.5.0 torchaudio 2.9.0 torchvision 0.24.0 xfuser 0.4.5 xgrammar 0.1.25	Ubuntu 24.04 Python 3.12 Torch 2.9.0+cu130 CUDA 13.0.2 diffusers 0.35.2 decord 0.6.0 decord2 2.0.0 flash_attn 2.8.3 flashinfer-python 0.5.2 imageio 2.37.2 imageio-ffmpeg 0.6.0 Ray 2.51.1 transformers 4.57.1 sgl-kernel 0.3.17.post1 sglang 0.5.5.post3 xgrammar 0.1.25 Triton 3.5.0 torchao 0.9.0 torchaudio 2.9.0 torchvision 0.24.0 xfuser 0.4.5 ljperf 0.1.0+d0e4a408	Ubuntu 24.04 Python 3.12 Torch 2.9.0+cu130 CUDA 13.0.2 diffusers 0.35.2 decord2 2.0.0 flashinfer-python 0.5.2 imageio 2.37.2 flash_attn 2.8.3 imageio-ffmpeg 0.6.0 Transformers 4.57.1 sgl-kernel 0.3.17.post1 sglang 0.5.5.post3 xgrammar 0.1.25 triton 3.5.0 torchao 0.9.0 torchaudio 2.9.0 torchvision 0.24.0 xfuser 0.4.5

Assets

Public images

CUDA 12.8 assets

egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:25.11-vllm0.11.1-pytorch2.9-cu128-20251120-serverless
egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:25.11-sglang0.5.5.post3-pytorch2.9-cu128-20251121-serverless

CUDA 13.0 assets

egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:25.11-vllm0.11.1-pytorch2.9-cu130-20251120-serverless
egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:25.11-sglang0.5.5.post3-pytorch2.9-cu130-20251121-serverless

VPC images

To pull ACS AI container images faster from within a VPC, replace the public image URI egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/{image:tag} with the VPC endpoint format acs-registry-vpc.{region-id}.cr.aliyuncs.com/egslingjun/{image:tag}.

{region-id}: The ID of the supported region for your ACS product. Examples: cn-beijing and cn-wulanchabu.
{image:tag}: The name and tag of the AI container image. Examples: inference-nv-pytorch:25.10-vllm0.11.0-pytorch2.8-cu128-20251028-serverless and training-nv-pytorch:25.10-serverless.

Note

These images are for ACS and multi-tenant Lingjun environments. They are not for single-tenant Lingjun environments.

Driver requirements

CUDA 12.8: NVIDIA driver release >= 570
CUDA 13.0: NVIDIA driver release >= 580

Quick start

The following example shows how to pull an inference-nv-pytorch image using Docker and test an inference service with the Qwen2.5-7B-Instruct model.

Note

To use an inference-nv-pytorch image in ACS, select it from the Artifact Center page in the console when you create a workload, or specify the image reference in a YAML file. For more information, see the following topics about building a model inference service using ACS GPU computing power:

Pull the inference container image.

docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

Download the open-source model from ModelScope.

pip install modelscope
cd /mnt
modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

Run the following command to start and enter the container.

docker run -it --rm --gpus all --network=host --privileged --init --ipc=host \
--ulimit memlock=-1 --ulimit stack=67108864  \
-v /mnt/:/mnt/ \
egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

Run a test to verify the vLLM chat inference feature.

Start the server.

python3 -m vllm.entrypoints.openai.api_server \
--model /mnt/Qwen2.5-7B-Instruct \
--trust-remote-code --disable-custom-all-reduce \
--tensor-parallel-size 1

Send a test request from the client.

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "/mnt/Qwen2.5-7B-Instruct",  
    "messages": [
    {"role": "system", "content": "You are a friendly AI assistant."},
    {"role": "user", "content": "Introduce deep learning."}
    ]}'

Output:

{"id":"chat-d3c28759793d4376a65bfc4e40b59a71","object":"chat.completion","created":1735278194,"model":"/mnt/deep_learning_test/testsuite/dataset/llms_inference_qwen7b-v2.5_accelerate/checkpoint/7B-V2.5/","choices":[{"index":0,"message":{"role":"assistant","content":"Deep learning is a branch of machine learning inspired by biological nervous systems, especially the interaction between neurons in the brain. It uses deep neural networks to process and analyze large amounts of data to identify effective prediction models. This technology has achieved significant success in many fields, such as image recognition, speech recognition, and natural language processing.\n\nIn deep learning, a neural network consists of multiple layers, including an input layer, several hidden layers, and an output layer. Each layer contains multiple nodes, or neurons, which are connected to nodes in other layers through weighted connections. During training, the neural network adjusts the weights of these connections based on the input data to minimize the error between the predicted output and the actual output. This process is typically implemented by using optimization algorithms such as gradient descent.\n\nThe training of deep learning models requires substantial computing resources and data. In recent years, with advances in computing hardware, such as GPUs and TPUs, and the rapid growth of datasets, deep learning technology has been widely adopted and developed. In addition to the applications mentioned above, deep learning is also extensively used in various fields, including medical diagnosis, autonomous driving, gaming, and finance.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":237,"completion_tokens":213}}

For more information about vLLM, see the vLLM documentation.

Known issues

The deepgpu-comfyui plug-in, which accelerates video generation for Wanx models, supports only the GN8IS, G49E, and G59 instance types.