inference-nv-pytorch 25.09

Main features and bug fixes

Contents

Asset

acs-registry-vpc.{region-id}.cr.aliyuncs.com/egslingjun/{image:tag}

Driver requirements

NVIDIA Driver release >= 570

Quick start

The following example shows how to pull the inference-nv-pytorch image with Docker and test the inference service with the Qwen2.5-7B-Instruct model.

1. Pull the inference container image.

   docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

2. Download the open-source model in ModelScope format.

   pip install modelscope
   cd /mnt
   modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

3. Run the following command to enter the container.

   docker run -it --rm --gpus all --network=host --privileged --init --ipc=host \
   --ulimit memlock=-1 --ulimit stack=67108864  \
   -v /mnt/:/mnt/ \
   egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

4. Run an inference test to verify vLLM's chat functionality.

   1. Start the server.

      python3 -m vllm.entrypoints.openai.api_server \
      --model /mnt/Qwen2.5-7B-Instruct \
      --trust-remote-code --disable-custom-all-reduce \
      --tensor-parallel-size 1

   2. Run a test on the client.

      curl http://localhost:8000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
          "model": "/mnt/Qwen2.5-7B-Instruct",  
          "messages": [
          {"role": "system", "content": "You are a friendly AI assistant."},
          {"role": "user", "content": "Tell me about deep learning."}
          ]}'

      Output:

      {"id":"chat-d3c28759793d4376a65bfc4e40b59a71","object":"chat.completion","created":1735278194,"model":"/mnt/deep_learning_test/testsuite/dataset/llms_inference_qwen7b-v2.5_accelerate/checkpoint/7B-V2.5/","choices":[{"index":0,"message":{"role":"assistant","content":"Deep learning is a branch of machine learning inspired by biological nervous systems, especially how neurons interact in the brain. It uses deep neural networks to process and analyze large amounts of data and to learn predictive models. This approach has achieved strong results in areas such as image recognition, speech recognition, and natural language processing.\n\nIn deep learning, a neural network consists of multiple layers, including an input layer, one or more hidden layers, and an output layer. Each layer contains multiple nodes (neurons). These nodes connect to nodes in other layers through weighted links. During training, the network adjusts these weights based on the input data to minimize the error between predicted outputs and actual results. This process often uses optimization algorithms such as gradient descent.\n\nTraining deep learning models often requires significant compute and large datasets. In recent years, advances in compute hardware (such as GPUs and TPUs) and the rapid growth of datasets have driven broad adoption and fast progress. Beyond the areas mentioned above, deep learning is also widely used in medical diagnosis, autonomous driving, gaming, and finance.","tool_calls":[],"logprobs":null,"finish_reason":"stop","stop_reason":null}}],"usage":{"prompt_tokens":237,"completion_tokens":213}}

      For more information on using vLLM, see the vLLM documentation.

Known issues

• The deepgpu-comfyui plugin, which accelerates video generation for Wanx models, only supports GN8IS and G49E instances.