DocsICP FilingConsole
官方文档
首页

inference-nv-pytorch 25.08

更新时间：
复制 MD 格式
产品详情
我的收藏

These are the release notes for inference-nv-pytorch 25.08.

Main features and bug fixes

Main features

  • Upgraded vLLM to v0.10.0.

  • Upgraded SGLang to v0.4.10.post2.

Bug fixes

None.

Contents

inference-nv-pytorch

inference-nv-pytorch

Tag

25.08-vllm0.10.0-pytorch2.7-cu128-20250811-serverless

25.08-sglang0.4.10.post2-pytorch2.7-cu128-20250808-serverless

Application scenario

Large model inference

Large model inference

Framework

pytorch

pytorch

Requirements

NVIDIA Driver release >= 570

NVIDIA Driver release >= 570

System components

  • Ubuntu 24.04

  • Python 3.12

  • Torch 2.7.1+cu128

  • CUDA 12.8

  • NCCL 2.27.5

  • diffusers 0.34.0

  • deepgpu-comfyui 1.1.7

  • deepgpu-torch 0.0.24+torch2.7.0cu128

  • flash_attn 2.8.2

  • imageio 2.37.0

  • imageio-ffmpeg 0.6.0

  • ray 2.48.0

  • transformers 4.55.0

  • triton 3.3.1

  • vllm 0.10.0

  • xformers 0.0.31

  • xfuser 0.4.4

  • xgrammar 0.1.21

  • Ubuntu 24.04

  • Python 3.12

  • Torch 2.7.1+cu128

  • CUDA 12.8

  • NCCL 2.27.5

  • diffusers 0.34.0

  • deepgpu-comfyui 1.1.7

  • deepgpu-torch 0.0.24+torch2.7.0cu128

  • flash_attn 2.8.2

  • flash_mla 1.0.0+41b611f

  • flashinfer-python 0.2.9rc2

  • imageio 2.37.0

  • imageio-ffmpeg 0.6.0

  • transformers 4.54.1

  • sgl-kernel 0.2.8

  • sglang 0.4.10.post2

  • xgrammar 0.1.22

  • triton 3.3.1

  • torchao 0.9.0

Assets

Public images

  • egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:25.08-vllm0.10.0-pytorch2.7-cu128-20250811-serverless

  • egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:25.08-sglang0.4.10.post2-pytorch2.7-cu128-20250808-serverless

VPC image

acs-registry-vpc.{region-id}.cr.aliyuncs.com/egslingjun/{image:tag}
Replace {region-id} with the region where your Alibaba Cloud Container Compute Service (ACS) is activated (for example, cn-beijing or cn-wulanchabu).
Replace {image:tag} with the name and tag of the image.
Note

The inference-nv-pytorch:25.03-vllm0.8.2-pytorch2.6-cu124-20250328-serverless and inference-nv-pytorch:25.03-sglang0.4.4.post1-pytorch2.5-cu124-20250327-serverless images are compatible with ACS and Lingjun multi-tenant deployments only. It is not compatible with Lingjun single-tenant deployments.

Note

The inference-nv-pytorch:25.08-vllm0.10.0-pytorch2.7-cu128-20250811-serverless and inference-nv-pytorch:25.08-sglang0.4.10.post2-pytorch2.7-cu128-20250808-serverless images are suitable for ACS and Lingjun multi-tenant deployments. They are not compatible with Lingjun single-tenant deployments.

Driver requirements

NVIDIA Driver release >= 570

Quick start

This example shows how to use Docker to pull the inference-nv-pytorch image and test its inference service with the Qwen2.5-7B-Instruct model.

Note

To use the inference-nv-pytorch image in ACS, you can select it from the Artifact Center page when you create a workload in the console, or specify the image in a YAML file. For more information, see the following topics:

  1. Pull the inference container image.

    docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

  2. Download the open-source model in ModelScope format.

    pip install modelscope
cd /mnt
modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

  3. Run the following command to start and enter the container.

    docker run -it --rm --gpus all --network=host --privileged --init --ipc=host \
--ulimit memlock=-1 --ulimit stack=67108864  \
-v /mnt/:/mnt/ \
egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

  4. Test the conversational inference feature of vLLM.

    1. Start the server.

      python3 -m vllm.entrypoints.openai.api_server \
--model /mnt/Qwen2.5-7B-Instruct \
--trust-remote-code --disable-custom-all-reduce \
--tensor-parallel-size 1

    2. Run a test on the client.

      curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",  
    "messages": [
    {"role": "system", "content": "You are a friendly AI assistant."},
    {"role": "user", "content": "Tell me about deep learning."}
    ]}'

      Output:

      {"id":"chat-d3c28759793d4376a65bfc4e40b59a71","object":"chat.completion","created":1735278194,"model":"/mnt/deep_learning_test/testsuite/dataset/llms_inference_qwen7b-v2.5_accelerate/checkpoint/7B-V2.5/","choices":[{"index":0,"message":{"role":"assistant","content":"深度学习是机器学习的一个分支，它受到生物神经系统，尤其是大脑中神经元之间交互的启发。深度学习利用深层神经网络处理并分析大量数据，以识别出有效的预测模型。这种技术在图像识别、语音识别、自然语言处理等多条多条上取得了量著的成功。\n\n在深度学习中，神经网络由多层组成，包括输入层、若干隐藏层和输出层。每一层都包含多个节点（或称为神经元），这些节点通过加权连接与其他层的节点相连。在训练过程中，神经网络会根据输入数据调整这些连接的权重，以最小化预测输出与实际输出之间的误差。这一过程通常使用梯度下降等优化算法实现。\n\n深度学习模型的训练需要大量的计算资源和数据支持。近年来，随着计算硬件的进步（如GPU、TPU等），以及数据集的快速增长，深度学习技术得到了广泛应用和发展。除了上述提到的应用外，深度学习还被广泛应用于医疗诊断、自动驾驶、游戏和金融等多个领域。","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":237,"completion_tokens":213}}

      For more information, see vLLM.

Known issues

  • The deepgpu-comfyui plug-in, which accelerates video generation for the Wanx model, currently supports only GN8IS and G49E.

Previous：inference-nv-pytorch 25.09Next：inference-nv-pytorch 25.07