Deploy the Qwen-7B-Chat model on an 8th-generation Intel instance

Background

Step 1: Create an ECS instance

1. Go to the instance creation page.

2. Follow the on-screen instructions to configure the parameters and create an ECS instance.

   The following parameters are important. For information about how to configure other parameters, see Customize and purchase an instance.

   • Instance: Qwen-7B-Chat requires more than 16 GiB of memory. To ensure stable model performance, select at least ecs.c8i.4xlarge (32 GiB memory) for the instance type.

   • Image: Alibaba Cloud Linux 3.2104 LTS 64-bit.

   • Public IP: Select Assign Public IPv4 Address. Set the billing method for bandwidth to Pay-By-Traffic and the peak bandwidth to 100 Mbps to accelerate model downloads.

     

   • System disk: The Qwen-7B-Chat model requires 60 GiB of storage for data downloading, conversion, and execution. We recommend setting the system disk size to 100 GiB to ensure that the model runs smoothly.

3. Add a security group rule.

   In the inbound direction of the security group for the ECS instance, add a rule to allow traffic on port 22 for SSH access and port 7860 for WebUI access. For detailed instructions, see Add a security group rule.

Step 2: Install the container environment

1. Connect to the ECS instance.

   For detailed instructions, see Connect to a Linux instance by using Workbench.

2. Install and start Docker.

   For detailed instructions, see Install and use Docker and Docker Compose.

3. Pull and run the Intel xFasterTransformer container.

   sudo docker pull registry.openanolis.cn/openanolis/xfastertransformer:1.7.3-23
   sudo docker run -it --name xFT -h xFT --privileged --shm-size=16g --network host -v /mnt:/mnt -w /mnt/xFasterTransformer registry.openanolis.cn/openanolis/xfastertransformer:1.7.3-23

   The following output indicates that the xFasterTransformer container is running.

   

   Important

   All subsequent operations must be performed within the container. If you exit the container, you can restart it and attach to its shell by running the following commands:

   sudo docker start xFT
   sudo docker exec -it xFT bash

4. (Optional) Update the xFasterTransformer scripts.

   The xFasterTransformer image already includes the scripts for the corresponding version. You can update to the latest test scripts.

   yum update -y
   yum install -y git
   cd /root/xFasterTransformer
   git pull

Step 3: Prepare the model data

1. In the container, install the required dependencies.

   yum update -y
   yum install -y wget git git-lfs vim tmux

2. Enable Git LFS.

   Git LFS is required to download the pre-trained model.

   git lfs install

3. Create and navigate to the model data directory.

   mkdir /mnt/data
   cd /mnt/data

4. Create a tmux session.

   tmux

   Important

   Downloading the pre-trained model can be time-consuming, and download times vary depending on network conditions. We recommend downloading the model within a tmux session to prevent the download from being interrupted by a lost SSH connection.

5. Download the Qwen-7B-Chat pre-trained model.

   git clone https://www.modelscope.cn/qwen/Qwen-7B-Chat.git /mnt/data/qwen-7b-chat

   Important

   After you run the git clone command, Git starts downloading the repository and all its large files through Git LFS. This process can take from several minutes to a few hours, depending on the model size and your network conditions. Wait for the download to complete.

   The following output indicates that the model download is complete.

   

6. Convert the model data.

   The downloaded model is in Hugging Face format and must be converted to the xFasterTransformer format. This conversion generates the model in the /mnt/data/qwen-7b-chat-xft directory.

   python -c 'import xfastertransformer as xft; xft.QwenConvert().convert("/mnt/data/qwen-7b-chat")'

   The following output indicates that the model conversion is complete.

   

   Note

   Different models require different Convert classes. xFasterTransformer supports the following model conversion classes:

   • LlamaConvert

   • ChatGLMConvert

   • ChatGLM2Convert

   • ChatGLM3Convert

   • OPTConvert

   • BaichuanConvert

   • QwenConvert

Step 4: Run the model for AI chat

cd /root/xFasterTransformer/examples/pytorch
OMP_NUM_THREADS=$(($(lscpu | grep "^CPU(s):" | awk '{print $NF}') / 2)) LD_PRELOAD=libiomp5.so numactl -C $(seq -s, 0 2 $(($(lscpu | grep "^CPU(s):" | awk '{print $NF}') - 2))) -m 0 python demo.py -t /mnt/data/qwen-7b-chat -m /mnt/data/qwen-7b-chat-xft -d bf16 --chat true

Benchmark model performance

The benchmark script uses dummy model data by default, so you do not need to download a model. Run the following command to benchmark the model's performance.

cd /root/xFasterTransformer/benchmark
XFT_CLOUD_ENV=1 bash run_benchmark.sh -m qwen-7b -d bf16 -bs 1 -in 32 -out 32 -i 10

You can adjust the following runtime parameters to test performance in specific scenarios:

• -d or --dtype: Specifies the model quantization type.

  • bf16 (default)

  • bf16_fp16

  • int8

  • bf16_int8

  • fp16

  • bf16_int4

  • int4

  • bf16_nf4

  • nf4

  • bf16_w8a8

  • w8a8

  • w8a8_int8

  • w8a8_int4

  • w8a8_nf4

• -bs or --batch_size: Specifies the batch size. The default value is 1.

• -in or --input_tokens: Specifies the input length. To use a custom length, configure the corresponding prompt in the prompt.json file. The default value is 32.

• -out or --output_tokens: Specifies the output length. The default value is 32.

• -i or --iter: Specifies the number of iterations. A larger number of iterations increases the test duration. The default value is 10.

Example output: