Deploy Qwen-72B-Chat on a c8i instance for AI conversation-Elastic Compute Service(ECS)-阿里云帮助中心

This topic shows how to deploy the Qwen-72B-Chat large language model on a single Alibaba Cloud 8th generation Intel instance using IPEX-LLM for AI conversation.

Background

What is Qwen-72B-Chat

Qwen-72B is a 72-billion-parameter model in the Qwen large language model series developed by Alibaba Cloud. The pretraining data for Qwen-72B is diverse and extensive, including a large amount of web text, professional books, and code. Qwen-72B-Chat is the chat-optimized, 72-billion-parameter model in the open-source Qwen series. It is created from Qwen-72B by using alignment techniques to follow human instructions.

What is IPEX-LLM

IPEX-LLM is an acceleration library optimized for large language models. It provides low-precision options like INT4, INT5, and INT8, and leverages Intel^® CPU hardware acceleration (AVX, VNNI, AMX) and the latest software optimizations. This enables faster execution of large language models on the Intel^® platform.

Important

The Qwen-72B-Chat model is open-sourced under the LICENSE. For free commercial use, you must fill out the Commercial License Application. You must comply with the user agreements, usage specifications, and relevant laws and regulations of third-party models. You are solely responsible for the legality and compliance of your use of third-party models.

Create an ECS instance

Go to the instance creation page.
Follow the on-screen instructions to create and configure an ECS instance.

Configure the following key parameters. For other parameters, see Create an instance by using the wizard.
- Instance: Qwen-72B-Chat requires approximately 70 GiB of memory. For stable model performance, we recommend an instance type with 192 GiB of memory or more. This topic uses ecs.c8i.24xlarge as an example.
- Image: Alibaba Cloud Linux 3.2104 LTS 64-bit.
- Public IP: Select Assign Public IPv4 Address. Set the network billing method to Pay-by-traffic and the peak bandwidth to 100 Mbps to accelerate model downloads.
- Data Disk: Running Qwen-72B-Chat requires downloading multiple model files, which consumes a large amount of storage space. To ensure the model runs smoothly, we recommend a 300 GiB data disk.
Add a security group rule.

In the instance's security group, add an inbound rule to allow traffic on ports 22, 443, and 7860 (for WebUI access). For more information, see Add a security group rule.
After the instance is created, find its public IP address on the ECS instance list page.

Note
The public IP address is used to access the WebUI service for AI conversation.

Deploy Qwen-72B-Chat

Step 1: Configure the runtime environment

Connect to the ECS instance.

For more information, see Connect to a Linux instance by using Workbench.
Install Docker.

For more information, see Install and use Docker and Docker Compose.

Run the following command to download the ipex-llm-serving-cpu image from Docker Hub.

Note

The ipex-llm-serving-cpu image includes the required environment, allowing you to run Qwen immediately after starting the container.

sudo docker pull intelanalytics/ipex-llm-serving-cpu:2.2.0-SNAPSHOT

[ecs-user@iZj6ceilixxx        ~]$ sudo docker pull intelanalytics/ipex-llm-serving-cpu:2.2.0-SNAPSHOT
2.2.0-SNAPSHOT: Pulling from intelanalytics/ipex-llm-serving-cpu
01007420e9b0: Pull complete
63341d78856e: Pull complete
5d0bb6adc43b: Pull complete
d0b97f902bf5: Pull complete
2d2f853f4f57: Pull complete
17d02779f97f: Pull complete
ca2aaaf805cb: Pull complete
c5b81183631f: Pull complete
febd299d716b: Pull complete
ce599ea20f5a: Pull complete
Digest: sha256:a71da0380feef4bb33426e9eb9c174c29be5a45b3fbd9xxx
Status: Downloaded newer image for intelanalytics/ipex-llm-serving-cpu:2.2.0-SNAPSHOT
docker.io/intelanalytics/ipex-llm-serving-cpu:2.2.0-SNAPSHOT

Run the following command to verify that the ipex-llm-serving-cpu image has been pulled.

sudo docker images

If the information returned by the system contains the ipex-llm-serving-cpu image, the image is downloaded successfully.

[ecs-user@iZj6ceiliafk  ~]$ sudo docker images
REPOSITORY                          TAG              IMAGE ID       CREATED        SIZE
intelanalytics/ipex-llm-serving-cpu  2.2.0-SNAPSHOT   bf6e756bda6e   17 hours ago   7.67GB

Step 2: Download the Qwen-72B-Chat pretrained model

Run the following command to install tmux:
```
sudo yum install tmux -y
```
Run the following command to create a tmux session window:
```
tmux
```
Important
Downloading the large pretrained model is time-consuming and network-dependent. We recommend running the download in a tmux session to prevent interruptions from a lost connection.
Use one of the following methods to download the Qwen-72B-Chat pretrained model.
Important
- Downloading via Git generates large intermediate files. To save disk space, we recommend using the Python script.
- With a 100 Mbps bandwidth, downloading the large Qwen-72B-Chat pretrained model takes about 3 hours. Please be patient.
Python script
1. Run the following command to download the miniconda installer.
```
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
```
2. Run the following command to install miniconda.
```
bash Miniconda3-latest-Linux-x86_64.sh
```
3. Run the following command to activate miniconda.
```
source ~/.bashrc
```
  If the command-line prompt starts with (base), miniconda is activated.
4. Run the following command to install pip.
```
sudo yum install -y python3-pip
```
  Run the following commands to upgrade the pip tool and the setuptools and wheel libraries.
```
sudo pip3 install --upgrade pip
sudo pip3 install --upgrade setuptools wheel
```
5. After miniconda is installed, install modelscope in the conda environment.
```
pip3 install modelscope
```
6. Run a Python script to download the Qwen-72B-Chat pretrained model.
  1. Run the following command to create a Python script file named test.py.
    sudo vim test.py
  2. Press i to enter insert mode and add the following content.
    from modelscope import snapshot_download model_dir = snapshot_download('qwen/Qwen-72B-Chat') print(model_dir)
  3. Press Esc to exit insert mode, and then enter :wq to save and exit.
  4. Run the following command to run the test.py Python script.
    python3 test.py
Git
1. Run the following command to install Git.
```
sudo yum install git -y
```
2. Download and install Git LFS to ensure that large files can be downloaded using Git commands.
```
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.rpm.sh | sudo bash
sudo yum install -y git-lfs
```
3. Run the following command to verify that Git LFS has been installed.
```
git lfs version
```
  If the version information of Git LFS is returned, Git LFS is installed.
4. Run the following command to download the Qwen-72B-Chat pretrained model.
```
sudo git clone https://www.modelscope.cn/qwen/Qwen-72B-Chat.git Qwen-72b-chat
```
Run the following command to view the directory path of the Qwen-72B-Chat pretrained model.
```
ls -l
```
```
drwxr-xr-x 4 root root        4096 Dec 11 11:42 Qwen-72B-Chat
```

Step 3: Start the IPEX-LLM container

Create a startup script file for the Docker container.
1. Run the following command to create and edit the start-container.sh script file.
```
sudo vim start-container.sh
```
2. Press i to enter insert mode and add the following information.
```
export MODEL_PATH=[YOUR_MODEL_PATH]
sudo docker run -itd \
        --net=host \
        --privileged \
        --cpuset-cpus="0-47" \
        --cpuset-mems="0" \
	  --name="qwen-llm" \
        --shm-size="16g" \
        -v $MODEL_PATH:/models \
        -e ENABLE_PERF_OUTPUT="true" \
        intelanalytics/ipex-llm-serving-cpu:2.2.0-SNAPSHOT
```
  Note
  The parameters are described as follows:
  - MODEL_PATH: Replace [YOUR_MODEL_PATH] with the absolute path to the Qwen-72B-Chat pretrained model specified in sub-step 3. of Step 2.
  - $MODEL_PATH: When the Docker container starts, the model on the Host is mounted to the /models directory of the container.
  - cpuset-cpus: Specifies the CPU cores to use.
  - cpuset-mems: Specifies the ID of the NUMA memory node to use.
  You can run the lscpu command to check the number of CPU cores and the NUMA memory node IDs.
3. Press Esc to exit insert mode, and then enter :wq to save and exit.
Run the following command to start the Docker container.
```
sudo bash start-container.sh
```

Run the following command to check the status of the Docker container.

sudo docker ps -a

If the output shows the following, the Docker container has started.

CONTAINER ID   IMAGE                                                   COMMAND               CREATED          STATUS           PORTS     NAMES
2525d2da34f5   intelanalytics/ipex-llm-serving-cpu:2.2.0-SNAPSHOT   "/opt/entrypoint.sh"   10 minutes ago   Up 10 minutes              qwen-llm

Run the following command to enter the Docker container.
```
sudo docker exec -it qwen-llm bash
```

Step 4: Start the AI conversation

Run the following commands in sequence to optimize performance.

Note
Before starting the conversation, set environment variables and optimize threading for better performance on Intel CPUs.
```
source ipex-llm-init -t
export OMP_NUM_THREADS=[number of CPU cores]   # This example uses 48 CPU cores. Replace [number of CPU cores] with the number of cores that you want to use.
```

Start a conversation with Qwen-72B-Chat.

Terminal

Run the following commands to start a terminal-based AI conversation with Qwen-72B-Chat.

cd /llm/portable-zip
python3 chat.py --model-path /models/Qwen-72B-Chat

Note

The Qwen-72B-Chat pretrained model is automatically converted to the int4 format when it is loaded.

root@host:/llm/portable-zip# python chat.py --model-path /models/Qwen-72B-Chat
Loading checkpoint shards: 100%|xxx| 19/19 [00:00<00:00, 30.52it/s]
2024-01-25 05:57:22,231 - INFO - Converting the current model to sym_int4 format
StartRecentKVCache: 4, 512
H: Why does the sun always rise in the east?
BigDL-LLM:  This is because the Earth rotates from west to east, so the sun appears to rise in the east and set in the west. In addition, the Earth's axis is tilted relative to the sun, so the locations of sunrise and sunset change with the seasons.

Browser

Run the following commands to start the WebUI service.

python3 -m fastchat.serve.controller &
python3 -m bigdl.llm.serving.model_worker --model-path /models/Qwen-72B-Chat --device cpu &
python3 -m fastchat.serve.gradio_web_server

Note

When the model worker is loaded, the system automatically quantizes the model to the int4 format. No manual conversion is required.

After you run the commands, the terminal displays startup logs. When the Running on local URL: http://0.0.0.0:7860 message appears, the WebUI service is started and can be accessed at this address.

2024-01-25 06:03:40 | INFO | gradio_web_server | args: Namespace(host='0.0.0.0', port=None, sha
re=False, controller_url='http://localhost:21001', concurrency_count=10, model_list_mode='once'
, moderate=False, add_chatgpt=False, add_claude=False, add_palm=False, register_openai_compatib
le_models=None, gradio_auth_path=None)
2024-01-25 06:03:40 | INFO | controller | Register a new worker: http://localhost:21002
2024-01-25 06:03:40 | INFO | stdout | INFO:     ::1:42722 - "POST /worker_get_status HTTP/1.1"
 200 OK
2024-01-25 06:03:40 | INFO | controller | Register done: http://localhost:21002, {'model_names'
: ['Qwen-72B-Chat'], 'speed': 1, 'queue_length': 0}
2024-01-25 06:03:40 | INFO | stdout | INFO:     ::1:52818 - "POST /refresh_all_workers HTTP/1.1
" 200 OK
2024-01-25 06:03:40 | INFO | stdout | INFO:     ::1:52828 - "POST /list_models HTTP/1.1" 200 OK
2024-01-25 06:03:40 | INFO | gradio_web_server | Models: ['Qwen-72B-Chat']
2024-01-25 06:03:40 | INFO | stdout | Running on local URL:  http://0.0.0.0:7860
2024-01-25 06:03:40 | INFO | stdout |
2024-01-25 06:03:40 | INFO | stdout | To create a public link, set `share=True` in `launch()`.

In your browser, navigate to http://<ECS public IP address>:7860.
Enter your prompt in the chat box to start the AI conversation.

The page displays the current model: Qwen-72B-Chat. Enter a question in the input box and click Send to generate a response.

Background

Create an ECS instance

Deploy Qwen-72B-Chat

Step 1: Configure the runtime environment

Step 2: Download the Qwen-72B-Chat pretrained model

Python script

Git

Step 3: Start the IPEX-LLM container

Step 4: Start the AI conversation

Terminal

Browser