Deploy DeepSeek-R1 distilled model on a GPU instance

Background

Step 1: Prepare the environment

1. Create a GPU instance and install the driver. For detailed instructions, see Create a GPU instance. Key parameters are described below.

   • Instance Type: See Estimate required configurations to select an instance type that meets the requirements for your chosen model version.

   • Images: Select a public image. This topic uses the Alibaba Cloud Linux 3.2104 LTS 64-bit image as an example.

     To deploy the DeepSeek-V3/R1 model on a GPU instance, you must install GPU driver version 550 or later. When purchasing the GPU instance from the ECS console, select the Install GPU Driver option to automatically install the required libraries (Tesla driver, CUDA, and cuDNN) upon instance creation. This method is faster than a manual installation.

     After selecting this option, the following versions are automatically installed: CUDA version 12.4.1, Driver version 550.127.08, and CUDNN version 9.2.0.82.

   • System Disk: See Estimate required configurations to set a system disk size that meets your model's requirements. We recommend a system disk size of at least 1 TiB.

   • Public IP Address: Select Assign Public IPv4 Address. For the bandwidth billing method, select Pay-by-traffic. To accelerate model downloads, we recommend setting the peak bandwidth to 100 Mbps.

   • Security Group: In your security group rules, open ports 22 and 8080.

2. Install Docker. For detailed instructions, see Install and use Docker and Docker Compose.

3. Install the NVIDIA Container Toolkit.

   If the installation fails, see the troubleshooting guide.

   Alibaba Cloud Linux/CentOS

   # Configure the production repository
   curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
     sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
   # Install the NVIDIA Container Toolkit package
   sudo yum install -y nvidia-container-toolkit
   # Restart Docker
   sudo systemctl restart docker

   Ubuntu/Debian

   # Configure the production repository
   curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
     && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
       sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
       sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
   # Update the package list from the repository
   sudo apt-get update
   # Install the NVIDIA Container Toolkit package
   sudo apt-get install -y nvidia-container-toolkit
   # Restart Docker
   sudo systemctl restart docker

4. (Optional) Purchase and mount a data disk.

   If your system disk lacks sufficient capacity for the large model files, purchase a separate data disk to store them. We recommend using /mnt as the mount point. For instructions, see Attach a data disk.

Step 2: Deploy and run the DeepSeek model

1. Run the following command to pull the inference image.

   sudo docker pull egs-registry.cn-hangzhou.cr.aliyuncs.com/egs/vllm:0.6.4.post1-pytorch2.5.1-cuda12.4-ubuntu22.04

2. Download the model files. You can visit the ModelScope community to select a model and find its name on the model details page.

   # Define the name of the model to download. Find the MODEL_NAME on the model details page in ModelScope. This script uses DeepSeek-R1-Distill-Qwen-7B as an example.
   MODEL_NAME="DeepSeek-R1-Distill-Qwen-7B"
   # Set the local storage path. Ensure this path has enough space for the model files (1.5 times the model size is recommended). This example uses /mnt/7B.
   LOCAL_SAVE_PATH="/mnt/7B"
   # If the /mnt/7B directory does not exist, create it.
   sudo mkdir -p ${LOCAL_SAVE_PATH}
   # Ensure the current user has write permissions for the directory. Adjust permissions as needed.
   sudo chmod ugo+rw ${LOCAL_SAVE_PATH}
   # Start the download. The container is automatically removed upon completion.
   sudo docker run -d -t --network=host --rm --name download \
   -v ${LOCAL_SAVE_PATH}:/data \
   egs-registry.cn-hangzhou.cr.aliyuncs.com/egs/vllm:0.6.4.post1-pytorch2.5.1-cuda12.4-ubuntu22.04 \
   /bin/bash -c "git-lfs clone https://www.modelscope.cn/models/deepseek-ai/${MODEL_NAME}.git /data"

3. Run the following command to monitor the download progress in real time. Wait for the download to finish.

   sudo docker logs -f download

   The model download may take a long time. Please be patient. When the download is complete, the log output will stop. You can press Ctrl+C at any time to exit the log view without stopping the download. The download will continue even if you close the terminal.

4. Start the model inference service.

   # Define the name of the model. Find the MODEL_NAME on the model details page in ModelScope. This script uses DeepSeek-R1-Distill-Qwen-7B as an example.
   MODEL_NAME="DeepSeek-R1-Distill-Qwen-7B"
   # Set the local storage path. Ensure this path has enough space for the model files (1.5 times the model size is recommended). This example uses /mnt/7B.
   LOCAL_SAVE_PATH="/mnt/7B"
   # Define the port for the service to listen on. You can adjust this as needed. The default port is 30000.
   PORT="30000"
   # Define the number of GPUs to use. This depends on the number of available GPUs on your instance, which you can check with the `nvidia-smi -L` command.
   # This example assumes the use of 4 GPUs.
   TENSOR_PARALLEL_SIZE="4"
   # Create the directory and set permissions. If the directory does not exist, this creates it and ensures the current user has read and write permissions.
   sudo mkdir -p ${LOCAL_SAVE_PATH}
   sudo chmod ugo+rw ${LOCAL_SAVE_PATH}
   # Start the Docker container and run the service.
   sudo docker run -d -t --network=host --gpus all \
       --privileged \
       --ipc=host \
       --name ${MODEL_NAME} \
       -v ${LOCAL_SAVE_PATH}:/data \
       egs-registry.cn-hangzhou.cr.aliyuncs.com/egs/vllm:0.6.4.post1-pytorch2.5.1-cuda12.4-ubuntu22.04 \
       /bin/bash -c "vllm serve /data \
           --port ${PORT} \
           --served-model-name ${MODEL_NAME} \
           --tensor-parallel-size ${TENSOR_PARALLEL_SIZE} \
           --max-model-len=16384 \
           --enforce-eager \
           --dtype=half"

5. Run the following command to check if the service started correctly.

   sudo docker logs ${MODEL_NAME}

   Look for a message in the log output similar to the following:

   INFO:     Uvicorn running on http://0.0.0.0:30000 (Press CTRL+C to quit)

   This indicates that the service has started successfully and is listening on port 30000.

Step 3: Start Open WebUI

1. Run the following command to pull the base environment image.

   sudo docker pull alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/python:3.11.1

2. Run the following command to start the Open WebUI service.

   # Set the model service address.
   OPENAI_API_BASE_URL=http://127.0.0.1:30000/v1
   # Create the data directory. Ensure the data directory exists and is located at /mnt.
   sudo mkdir -p /mnt/open-webui-data
   # Start the Open WebUI service.
   sudo docker run -d -t --network=host --name open-webui \
   -e ENABLE_OLLAMA_API=False \
   -e OPENAI_API_BASE_URL=${OPENAI_API_BASE_URL} \
   -e DATA_DIR=/mnt/open-webui-data \
   -e HF_HUB_OFFLINE=1 \
   -v /mnt/open-webui-data:/mnt/open-webui-data \
   alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/python:3.11.1 \
   /bin/bash -c "pip config set global.index-url http://mirrors.cloud.aliyuncs.com/pypi/simple/ && \
   pip config set install.trusted-host mirrors.cloud.aliyuncs.com && \
   pip install --upgrade pip && \
   pip install open-webui==0.5.10 && \
   mkdir -p /usr/local/lib/python3.11/site-packages/google/colab && \
   open-webui serve"

3. Run the following command to monitor the service startup process in real time.

   sudo docker logs -f open-webui

   Look for a message in the log output similar to the following:

   INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)

   This indicates that the service has started successfully and is listening on port 8080.

4. On your local machine, open a web browser and navigate to http://<ECS public IP address>:8080. When you first visit, a setup form appears. Enter your Name, Email, and Password, then click Create Admin Account to register your administrator account.

5. Test the question-and-answer functionality in the Open WebUI interface.

   The interface displays the current model, DeepSeek-R1-Distill-Qwen-7B, at the top. When you ask a question in the chat area, the model first shows its Chain of Thought process in orange text before providing the final answer. Below the chat, you can find an input box, a file upload option, and the code interpreter.

6. If you want to allow other users to access your Open WebUI deployment, follow these steps.

   1. Click your avatar in the upper-right corner and select admin panel.

   2. In the admin panel, click Settings in the top menu bar.

   3. In the left navigation pane, click Common. Enable the Allow New User Sign-ups toggle, and set the Default User Role to User.

      In the settings panel, the Allow new user registration switch is enabled, and the Default user role drop-down list is configured.

   4. In the left navigation pane, click Models. Click the button next to the model.

   5. From the visibility dropdown list, select Public and click Save and Update.

   6. After completing the configuration, new users can visit http://<ECS public IP address>:8080 to sign up and log in to your Open WebUI deployment and interact with the DeepSeek-R1 distilled model.

Related operations

For log collection, see Collect standard output from Docker containers (legacy).