Deploy a distilled DeepSeek-R1 model on a GPU-accelerated instance

Background information

Step 1: Prepare the environment

1. Create a GPU-accelerated instance and install the driver. For more information, see Create a GPU-accelerated instance. The key parameters are described as follows.

   • Instance Type: Refer to Estimate required configurations and select an instance type that meets the requirements of your model version.

   • Images: Select a public image. This topic uses an Alibaba Cloud Linux 3.2104 LTS 64-bit image as an example.

     To deploy a DeepSeek-V3/R1 model on a GPU-accelerated instance, you must install GPU driver version 550 or later on the instance. When you purchase a GPU-accelerated instance in the ECS console, select Install GPU Driver. After the instance is created, the Tesla driver, CUDA, and cuDNN libraries are automatically installed. This method is faster than manual installation.

   • System Disk: Refer to Estimate required configurations and set the system disk size to meet the requirements of your model version. Set the system disk size to at least 1 TiB.

   • Public IP Address: Select Assign Public IPv4 Address. Set Billing Method for Bandwidth to Pay-by-traffic. Set the peak bandwidth to 100 Mbps to speed up model downloads.

   • Security Group: Open ports 22 and 8080.

2. Install Docker. For more information, see Install and use Docker and Docker Compose.

3. Install the NVIDIA Container Toolkit.

   If the installation fails, see Solutions.

   Alibaba Cloud Linux/CentOS

   # Configure the production repository
   curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
     sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
   # Install the NVIDIA Container Toolkit package
   sudo yum install -y nvidia-container-toolkit
   # Restart Docker
   sudo systemctl restart docker

   Ubuntu/Debian

   # Configure the production repository
   curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
     && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
       sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
       sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
   # Update the package list from the repository
   sudo apt-get update
   # Install the NVIDIA Container Toolkit package
   sudo apt-get install -y nvidia-container-toolkit
   # Restart Docker
   sudo systemctl restart docker

4. (Optional) Purchase and attach a data disk.

   The model file is large. If your system disk has insufficient capacity, you can purchase and attach a separate data disk to store the downloaded model. Use /mnt as the mount target. For more information, see Attach a data disk.

Step 2: Deploy and run the DeepSeek model

1. Run the following command to pull the inference image.

   sudo docker pull egs-registry.cn-hangzhou.cr.aliyuncs.com/egs/vllm:0.6.4.post1-pytorch2.5.1-cuda12.4-ubuntu22.04

2. Download the model file. You can visit ModelScope, the Alibaba Cloud model community, to select a model and obtain its name from the model's product page.

   # Define the name of the model to download. You need to visit ModelScope to select a model and get its name from the model's product page. This script uses DeepSeek-R1-Distill-Qwen-7B as an example.
   MODEL_NAME="DeepSeek-R1-Distill-Qwen-7B"
   # Set the local storage path. Make sure this path has enough space for the model file (reserve 1.5 times the model size). This example uses /mnt/7B.
   LOCAL_SAVE_PATH="/mnt/7B"
   
   # If the /mnt/7B folder does not exist, create it.
   sudo mkdir -p ${LOCAL_SAVE_PATH}
   # Make sure the current user has write permissions for this folder. Adjust the permissions as needed.
   sudo chmod ugo+rw ${LOCAL_SAVE_PATH}
   
   # Start the download. The container is automatically destroyed after the download is complete.
   sudo docker run -d -t --network=host --rm --name download \
   -v ${LOCAL_SAVE_PATH}:/data \
   egs-registry.cn-hangzhou.cr.aliyuncs.com/egs/vllm:0.6.4.post1-pytorch2.5.1-cuda12.4-ubuntu22.04 \
   /bin/bash -c "git-lfs clone https://www.modelscope.cn/models/deepseek-ai/${MODEL_NAME}.git /data"

3. Run the following command to monitor the download progress in real time. Wait for the download to finish.

   sudo docker logs -f download

   The model download takes a long time. When the download task is complete, no new logs appear. You can press Ctrl+C to exit at any time. This action does not affect the container's operation. The download continues even if you exit the terminal.

4. Start the model inference service.

   # Define the name of the model to download. You need to visit ModelScope to select a model and get its name from the model's product page. This script uses DeepSeek-R1-Distill-Qwen-7B as an example.
   MODEL_NAME="DeepSeek-R1-Distill-Qwen-7B"
   # Set the local storage path. Make sure this path has enough space for the model file (reserve 1.5 times the model size). This example uses /mnt/7B.
   LOCAL_SAVE_PATH="/mnt/7B"
   
   # Define the port number for the service runtime to listen on. You can adjust this as needed. The default port is 30000.
   PORT="30000"
   
   # Define the number of GPUs to use. This depends on the number of available GPUs on the instance. You can query this with the nvidia-smi -L command.
   # This example assumes four GPUs are used.
   TENSOR_PARALLEL_SIZE="4"
   
   # Create the folder and set permissions. If the folder does not exist, create it and make sure the current user has read and write permissions.
   sudo mkdir -p ${LOCAL_SAVE_PATH}
   sudo chmod ugo+rw ${LOCAL_SAVE_PATH}
   
   # Start the Docker container and run the service.
   sudo docker run -d -t --network=host --gpus all \
       --privileged \
       --ipc=host \
       --name ${MODEL_NAME} \
       -v ${LOCAL_SAVE_PATH}:/data \
       egs-registry.cn-hangzhou.cr.aliyuncs.com/egs/vllm:0.6.4.post1-pytorch2.5.1-cuda12.4-ubuntu22.04 \
       /bin/bash -c "vllm serve /data \
           --port ${PORT} \
           --served-model-name ${MODEL_NAME} \
           --tensor-parallel-size ${TENSOR_PARALLEL_SIZE} \
           --max-model-len=16384 \
           --enforce-eager \
           --dtype=half"

5. Run the following command to check whether the service started correctly.

   sudo docker logs ${MODEL_NAME}

   A message similar to the following appears in the log output:

   INFO:     Uvicorn running on http://0.0.0.0:30000 (Press CTRL+C to quit)

   This message indicates that the service has started successfully and is listening on port 30000.

Step 3: Start Open WebUI

1. Run the following command to pull the base environment image.

   sudo docker pull alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/python:3.11.1

2. Run the following command to start the Open WebUI service.

   # Set the model service endpoint.
   OPENAI_API_BASE_URL=http://127.0.0.1:30000/v1
   
   # Create a data folder. Make sure the data folder exists under /mnt.
   sudo mkdir -p /mnt/open-webui-data
   
   # Start the open-webui service.
   sudo docker run -d -t --network=host --name open-webui \
   -e ENABLE_OLLAMA_API=False \
   -e OPENAI_API_BASE_URL=${OPENAI_API_BASE_URL} \
   -e DATA_DIR=/mnt/open-webui-data \
   -e HF_HUB_OFFLINE=1 \
   -v /mnt/open-webui-data:/mnt/open-webui-data \
   alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/python:3.11.1 \
   /bin/bash -c "pip config set global.index-url http://mirrors.cloud.aliyuncs.com/pypi/simple/ && \
   pip config set install.trusted-host mirrors.cloud.aliyuncs.com && \
   pip install --upgrade pip && \
   pip install open-webui==0.5.10 && \
   mkdir -p /usr/local/lib/python3.11/site-packages/google/colab && \
   open-webui serve"

3. Run the following command to monitor the download progress in real time. Wait for the download to finish.

   sudo docker logs -f open-webui

   A message similar to the following appears in the log output:

   INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)

   This message indicates that the service has started successfully and is listening on port 8080.

4. On your local physical machine, use a browser to access http://<Public IP address of the ECS instance>:8080. When you log on for the first time, follow the prompts to create an administrator account.

5. Test the Q&A feature in the Open WebUI interface.

6. If you want to open your deployed Open WebUI to regular users, perform the following steps.

   1. Click the profile picture in the upper-right corner and select Admin Panel.

   2. Go to the admin backend. In the top menu bar, click Settings.

   3. In the navigation pane on the left, click General. Turn on the Allow New User Sign-ups switch. Set Default User Role to user.

   4. In the navigation pane on the left, click Models. Click the

   5. From the visibility drop-down list, select Public. Click Save and Update.

   6. After the configuration is complete, new users can access http://<Public IP address of the ECS instance>:8080 to register and log on to your deployed Open WebUI to have conversations with the DeepSeek-R1-Distill model.

Related operations

For log collection, see Collect the standard output of a Docker container (legacy).