inference-nv-pytorch 25.11

Key features and bug fixes

Contents

Quick start

The following example shows how to pull an inference-nv-pytorch image using Docker and test an inference service with the Qwen2.5-7B-Instruct model.

1. Pull the inference container image.

   docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

2. Download the open-source model from ModelScope.

   pip install modelscope
   cd /mnt
   modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

3. Run the following command to start and enter the container.

   docker run -it --rm --gpus all --network=host --privileged --init --ipc=host \
   --ulimit memlock=-1 --ulimit stack=67108864  \
   -v /mnt/:/mnt/ \
   egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

4. Run a test to verify the vLLM chat inference feature.

   1. Start the server.

      python3 -m vllm.entrypoints.openai.api_server \
      --model /mnt/Qwen2.5-7B-Instruct \
      --trust-remote-code --disable-custom-all-reduce \
      --tensor-parallel-size 1

   2. Send a test request from the client.

      curl http://localhost:8000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
          "model": "/mnt/Qwen2.5-7B-Instruct",  
          "messages": [
          {"role": "system", "content": "You are a friendly AI assistant."},
          {"role": "user", "content": "Introduce deep learning."}
          ]}'

      Output:

      {"id":"chat-d3c28759793d4376a65bfc4e40b59a71","object":"chat.completion","created":1735278194,"model":"/mnt/deep_learning_test/testsuite/dataset/llms_inference_qwen7b-v2.5_accelerate/checkpoint/7B-V2.5/","choices":[{"index":0,"message":{"role":"assistant","content":"Deep learning is a branch of machine learning inspired by biological nervous systems, especially the interaction between neurons in the brain. It uses deep neural networks to process and analyze large amounts of data to identify effective prediction models. This technology has achieved significant success in many fields, such as image recognition, speech recognition, and natural language processing.\n\nIn deep learning, a neural network consists of multiple layers, including an input layer, several hidden layers, and an output layer. Each layer contains multiple nodes, or neurons, which are connected to nodes in other layers through weighted connections. During training, the neural network adjusts the weights of these connections based on the input data to minimize the error between the predicted output and the actual output. This process is typically implemented by using optimization algorithms such as gradient descent.\n\nThe training of deep learning models requires substantial computing resources and data. In recent years, with advances in computing hardware, such as GPUs and TPUs, and the rapid growth of datasets, deep learning technology has been widely adopted and developed. In addition to the applications mentioned above, deep learning is also extensively used in various fields, including medical diagnosis, autonomous driving, gaming, and finance.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":237,"completion_tokens":213}}

      For more information about vLLM, see the vLLM documentation.

Known issues

• The deepgpu-comfyui plug-in, which accelerates video generation for Wanx models, supports only the GN8IS, G49E, and G59 instance types.