Deploy the full-featured DeepSeek model on two distributed GPU-accelerated instances

Procedure

Step 3: Test the model operation

1. Enter the vllm-test container.

   docker exec -it vllm-test bash

2. Set GLOO to use the eth0 network interface card.

   This step must be performed on both node1 and node2.

   echo 'export GLOO_SOCKET_IFNAME=eth0' >> ~/.bashrc
   echo 'export NCCL_SOCKET_IFNAME=eth0' >> ~/.bashrc
   source ~/.bashrc

3. Install Ray and create a Ray cluster between the two machines.

   pip3 install ray
   
   # Run on node1
   ray start --head --dashboard-host 0.0.0.0

   On node1, output similar to the following is displayed.

   root@xxx /workspace# ray start --head --dashboard-host 0.0.0.0
   Enable usage stats collection? This prompt will auto-proceed in 10 seconds to avoid blocking cluster startup. Confirm [Y/n]: y
   Usage stats collection is enabled. To disable this, add `--disable-usage-stats` to the command that starts the cluster, or run the following
   command: `ray disable-usage-stats` before starting the cluster. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details.
   
   Local node IP: 1xxx
   
   --------------------
   Ray runtime started.
   --------------------
   
   Next steps
     To add another node to this Ray cluster, run
       ray start --address='xxx'
   
     To connect to this Ray cluster:
       import ray
       ray.init()
   
     To submit a Ray job using the Ray Jobs CLI:
       RAY_ADDRESS='xxx' xxx
   
     See https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html
     for more information on submitting Ray jobs to the Ray cluster.
   
     To terminate the Ray runtime, run
       ray stop
   
     To view the status of the cluster, use
       ray status
   
     To monitor and debug Ray, view the dashboard at
       xxx
   
     If connection to the dashboard fails, check your firewall settings and network configuration.

   The ray start --address='xxx' command shown above is the command to execute on node2 to join this Ray cluster.

4. Then, on node2, run the ray start --address='xxx' command shown in the preceding output to create the Ray cluster. You can run the ray status command on either machine to verify that the cluster is created successfully. The following output indicates that the cluster is created successfully.

   root@ixxx:/workspace# ray status
   ======== Autoscaler status: 2025-02-13 09:17:16.683342 ========
   Node status
   ---------------------------------------------------------------
   Active:
    1 node_dxxx3a8f
    1 node_8xxx50d7
   Pending:
    (no pending nodes)
   Recent failures:
    (no failures)
   
   Resources
   ---------------------------------------------------------------
   Usage:
    0.0/384.0 CPU
    0.0/16.0 GPU
    0B/1.59TiB memory
    0B/372.53GiB object_store_memory
   
   Demands:
    (no resource demands)

5. On either machine, run the following command in the container to start the inference service for DeepSeek-V3 or DeepSeek-R1.

   vllm serve /path/to/DeepSeek-V3 \
   --tensor-parallel-size 8 \
   --pipeline-parallel-size 2 \
   --trust-remote-code \
   --enable-chunked-prefill \
   --host 0.0.0.0

   Replace /path/to/DeepSeek-V3 with the path to the local storage folder for DeepSeek-V3 or DeepSeek-R1.

   The following output indicates that the vLLM inference service has started successfully.

   INFO 02-13 09:39:05 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s

6. Wait for the inference server to initialize. Then, open another terminal window in the container on the same machine, send an inference request, and view the model's response.

   curl http://localhost:8000/v1/chat/completions \
   -H "Content-Type:application/json" \
   -d '{
     "model":"/path/to/DeepSeek-V3",
     "messages":[{"role": "user", "content": "Write a 20-character poem about the moon"}],
     "stream":false}'

   root@xxx# curl http://localhost:8000/v1/chat/completions -H "Content-Type:application/json" -d '{"model":"/mnt/DeepSeek-R1","messages":[{"role": "user", "content": "Write a 20-character poem about the moon"}],"stream":false}' 
   {"id":"chatcmpl-a89681c0b63c4b38ba9e0fbfb3a4016e","object":"chat.completion","created":1739439522,"model":"/mnt/DeepSeek-R1","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"<think>\nOf course, here's the English translation of your thought process and the poem, retaining the requested tags:Okay, the user asked me to write a 20-character poem about the moon. First, I need to determine the poem's structure. For 20 characters, a five-character quatrain is quite suitable, with five characters per line and four lines making exactly 20 characters. Next, I need to consider imagery associated with the moon, such as the common jade disk, bright mirror, cassia tree, etc., but I need to avoid too common metaphors and aim for some novelty."},"tool_calls":[]}],"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":9,"total_tokens":459,"completion_tokens":450,"prompt_tok

(Optional) DeepSeek-V3/R1 model performance evaluation

To avoid downloading the evaluation dataset for each run, download it in advance and store it locally on the server.

1. In the container, install the software required for performance evaluation.

   apt install -y unzip
   pip install datasets -i https://mirrors.aliyun.com/pypi/simple

2. Clone the ShareGPT_V3_unfiltered_cleaned_split dataset repository.

   git lfs clone https://www.modelscope.cn/datasets/gliang1001/ShareGPT_V3_unfiltered_cleaned_split.git

3. Download the vLLM source code and decompress it.

   wget https://help-static-aliyun-doc.aliyuncs.com/install-script/vllm-0.7.2.zip
   unzip vllm-0.7.2.zip

4. Enable performance evaluation in serving mode.

   python3 ./vllm-0.7.2/benchmarks/benchmark_serving.py \
     --backend vllm \
     --model /path/to/DeepSeek-V3 \
     --dataset-name random \
     --random-output-len 4096 \
     --random-input-len 512 \
     --num-prompts 10 \
     --max-concurrency 1 \
     --port 8000 \
     --dataset-path ShareGPT_V3_unfiltered_cleaned_split/ShareGPT_V3_unfiltered_cleaned_split.json

   Replace /path/to/DeepSeek-V3 with the path to the local storage folder for DeepSeek-V3 or DeepSeek-R1.

5. View the evaluation results.

   ============ Serving Benchmark Result =============
   Successful requests:                     xxx
   Benchmark duration (s):                  xxx
   Total input tokens:                      xxx
   Total generated tokens:                  xxx
   Request throughput (req/s):              xxx
   Output token throughput (tok/s):         xxx
   Total Token throughput (tok/s):          xxx
   ----------------Time to First Token----------------
   Mean TTFT (ms):                          xxx
   Median TTFT (ms):                        xxx
   P99 TTFT (ms):                           xxx
   -----Time per Output Token (excl. 1st token)------
   Mean TPOT (ms):                          xxx
   Median TPOT (ms):                        xxx
   P99 TPOT (ms):                           xxx
   ----------------Inter-token Latency----------------
   Mean ITL (ms):                           xxx
   Median ITL (ms):                         xxx
   P99 ITL (ms):                            xxx
   ===================================================

References

• Compute Nest one-click deployment (dual-GPU-instance version)

  Compute Nest uses a ROS template to enable one-click deployment of cloud resources and large models. You only need to specify a few parameters when you create the service instance. You can deploy and use the full-featured DeepSeek model on two GPU-accelerated instances in under 30 minutes to experience the inference performance of DeepSeek-R1/V3. Click the deployment link to start the one-click deployment.

• Command-line deployment (single-GPU-instance version)

  You can use SGLang as the inference framework for the DeepSeek model and use commands to deploy the full-featured DeepSeek model on a single GPU-accelerated instance. This method also lets you experience the inference performance of DeepSeek-R1/V3 with no extra configuration.