inference-nv-pytorch 25.02

Main features and bug fixes

Contents

Assets

acs-registry-vpc.{region-id}.cr.aliyuncs.com/egslingjun/{image:tag}

Driver requirements

NVIDIA Driver release >= 550

Quick start

This example shows how to pull the inference-nv-pytorch image with Docker and test the inference service on the Qwen2.5-7B-Instruct model.

1. Pull the inference container image.

   docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

2. Download the open-source model in ModelScope format.

   pip install modelscope
   cd /mnt
   modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

3. Run the following command to start and enter the container.

   docker run -it --rm --gpus all --network=host --privileged --init --ipc=host \
   --ulimit memlock=-1 --ulimit stack=67108864  \
   -v /mnt/:/mnt/ \
   egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

4. Test the vLLM conversational inference feature.

   1. Start the server.

      python3 -m vllm.entrypoints.openai.api_server \
      --model /mnt/Qwen2.5-7B-Instruct \
      --trust-remote-code --disable-custom-all-reduce \
      --tensor-parallel-size 1

   2. Run a test from the client.

      curl http://localhost:8000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
          "model": "/mnt/Qwen2.5-7B-Instruct",  
          "messages": [
          {"role": "system", "content": "You are a friendly AI assistant."},
          {"role": "user", "content": "Please introduce deep learning."}
          ]}'

      The output is as follows:

      {"id":"chat-d3c28759793d4376a65bfc4e40b59a71","object":"chat.completion","created":1735278194,"model":"/mnt/deep_learning_test/testsuite/dataset/llms_inference_qwen7b-v2.5_accelerate/checkpoint/7B-V2.5/","choices":[{"index":0,"message":{"role":"assistant","content":"Deep learning is a subfield of machine learning inspired by biological nervous systems, particularly the interaction between neurons in the brain. Deep learning uses deep neural networks to process and analyze large amounts of data to identify effective predictive models. This technology has achieved significant success in numerous fields, such as image recognition, speech recognition, and natural language processing.\n\nIn deep learning, a neural network consists of multiple layers, including an input layer, several hidden layers, and an output layer. Each layer contains multiple nodes (or neurons), which are connected to nodes in other layers through weighted connections. During the training process, the neural network adjusts the weights of these connections based on the input data to minimize the error between the predicted output and the actual output. This process is typically accomplished using optimization algorithms like gradient descent.\n\nTraining deep learning models requires substantial computing resources and data. In recent years, with advancements in computing hardware (such as GPUs and TPUs) and the rapid growth of datasets, deep learning technology has been widely applied and developed. In addition to the applications mentioned above, deep learning is also extensively used in various other fields, including medical diagnostics, autonomous driving, gaming, and finance.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":237,"completion_tokens":213}}

      For more information about using vLLM, see vLLM.

Known issues

• Illegal memory access for MoE On H20 #13693. We recommend updating vLLM.