Deploy Qwen-7B-Chat using an 8th-generation Intel instance

Background information

Step 1: Create an ECS instance

1. Go to the instance creation page.

2. Follow the on-screen instructions to configure the parameters and create an ECS instance.

   Note the following parameters. For more information about other parameters, see Create an instance using the wizard.

   • Instance: Qwen-7B-Chat requires more than 16 GiB of memory to run. To ensure that the model runs stably, select at least ecs.c8i.4xlarge (32 GiB of memory) for the instance type.

   • Image: Alibaba Cloud Linux 3.2104 LTS 64-bit.

   • Public IP: Select Assign Public IPv4 Address. Set Billing Method for Bandwidth to Pay-By-Traffic. Set the peak bandwidth to 100 Mbps. This speeds up the model download.

   • System Disk: Downloading, converting, and running the Qwen-7B-Chat model data requires 60 GiB of storage space. To ensure that the model runs smoothly, set the system disk size to 100 GiB.

3. Add a security group rule.

   In the inbound direction of the security group for the ECS instance, add a rule to allow traffic on port 22 and port 7860. Port 22 is for the Secure Shell Protocol (SSH) service, and port 7860 is for the WebUI page. For more information, see Add a security group rule.

Step 2: Install the container environment for the model

1. Remotely connect to the ECS instance.

   For more information, see Log on to a Linux instance using Workbench.

2. Install and start Docker.

   For more information, see Install and use Docker and Docker Compose.

3. Pull and run the Intel xFasterTransformer container.

   sudo docker pull registry.openanolis.cn/openanolis/xfastertransformer:1.7.3-23
   sudo docker run -it --name xFT -h xFT --privileged --shm-size=16g --network host -v /mnt:/mnt -w /mnt/xFasterTransformer registry.openanolis.cn/openanolis/xfastertransformer:1.7.3-23

   If a message similar to the following one appears, the xFasterTransformer container has been pulled and is running.

   [ecs-user@iZuf6epjbtjc          ~]$ sudo docker pull registry.openanolis.cn/openanolis/xfastertransformer:1.7.3-23
   1.7.3-23: Pulling from openanolis/xfastertransformer
   a8ede12a3939: Pull complete
   9999739bce1f: Pull complete
   cb62f821a6d8: Pull complete
   ac8ed0e35b17: Pull complete
   5183b0b1eb1d: Pull complete
   375bd19cec22: Pull complete
   b0fec068aed0: Pull complete
   ebf0c1febfdb: Pull complete
   317c71eed5ce: Pull complete
   36095701a3ad: Pull complete
   f1b77b925530: Pull complete
   4f4fb700ef54: Pull complete
   4d755c4a190d: Pull complete
   Digest: sha256:e88c4b325a824046ff5d57402aaeb896bec2d129925e960ac93
   Status: Downloaded newer image for registry.openanolis.cn/openanolis/xfastertransformer:1.7.3-23
   registry.openanolis.cn/openanolis/xfastertransformer:1.7.3-23
   [ecs-user@iZuf6epjbtj          ~]$ sudo docker run -it --name xFT -h xFT --privileged --shm-size=16g --network host -v /mnt:/mnt -w /mnt/xFasterTransformer registry.openanolis.cn/openanolis/xfastertransformer:1.7.3-23
   
   :: initializing oneAPI environment ...
      bash: BASH_VERSION = 5.2.15(1)-release
      args: Using "$@" for setvars.sh arguments:
   :: ccl -- latest
   :: compiler -- latest
   :: mpi -- latest
   :: tbb -- latest
   
   :: oneAPI environment initialized ::
   
   [root@xFT xFasterTransformer]#

   Important

   The following operations must be run in the container. If you exit the container, you can run the following commands to start it and re-enter the container's shell environment.

   sudo docker start xFT
   sudo docker exec -it xFT bash

4. (Optional) Update the xFasterTransformer script code.

   The xFasterTransformer image already contains the script code for the corresponding version. You can update it to the latest test script.

   yum update -y
   yum install -y git
   cd /root/xFasterTransformer
   git pull

Step 3: Prepare the model data

1. Install dependencies in the container.

   yum update -y
   yum install -y wget git git-lfs vim tmux

2. Enable Git LFS.

   Git LFS is required to download the pre-trained model.

   git lfs install

3. Create and navigate to the model data directory.

   mkdir /mnt/data
   cd /mnt/data

4. Create a tmux session.

   tmux

   Important

   Downloading the pre-trained model takes a long time, and the success rate depends on network conditions. Download the model in a tmux session to prevent the download from being interrupted by a lost connection to the ECS instance.

5. Download the Qwen-7B-Chat pre-trained model.

   git clone https://www.modelscope.cn/qwen/Qwen-7B-Chat.git /mnt/data/qwen-7b-chat

   Important

   After you run the git clone command, Git starts to download the repository and all its large files through Git LFS. This process can take from several minutes to several hours, depending on the model size and network conditions.

   If the following message appears, the pre-trained model has been downloaded.

   [root@xFT /]# git clone https://www.modelscope.cn/qwen/Qwen-7B-Chat.git /mnt/data/qwen-7b-chat
   Cloning into '/mnt/data/qwen-7b-chat'...
   remote: Enumerating objects: 551, done.
   remote: Counting objects: 100% (53/53), done.
   remote: Compressing objects: 100% (27/27), done.
   remote: Total 551 (delta 28), reused 50 (delta 26), pack-reused 498
   Receiving objects: 100% (551/551), 16.47 MiB | 21.62 MiB/s, done.
   Resolving deltas: 100% (292/292), done.
   Filtering content: 100% (8/8), 14.38 GiB | 11.14 MiB/s, done.
   [root@xFT /]#

6. Convert the model data.

   The downloaded model data is in HuggingFace format and must be converted to the xFasterTransformer format. The generated model folder is /mnt/data/qwen-7b-chat-xft.

   python -c 'import xfastertransformer as xft; xft.QwenConvert().convert("/mnt/data/qwen-7b-chat")'

   If the following message appears, the model has been converted.

   [root@xFT /]# python -c 'import xfastertransformer as xft; xft.QwenConvert().convert("/mnt/data/qwen-7b-chat")'
   Processing ...
   Found 8 model parts: ['model-00001-of-00008.safetensors', 'model-00002-of-00008.safetensors', 'model-00003-of-00008.safetensors', 'model-00004-of-00008.safetensors', 'model-00005-of-00008.safetensors', 'model-00006-of-00008.safetensors', 'model-00007-of-00008.safetensors', 'model-00008-of-00008.safetensors']
   /mnt/data/qwen-7b-chat-xft export successful!
   [root@xFT /]#

   Note

   Different model data use different Convert classes. xFasterTransformer supports the following model conversion classes:

   • LlamaConvert

   • ChatGLMConvert

   • ChatGLM2Convert

   • ChatGLM3Convert

   • OPTConvert

   • BaichuanConvert

   • QwenConvert

Step 4: Run the model for AI chat

cd /root/xFasterTransformer/examples/pytorch
OMP_NUM_THREADS=$(($(lscpu | grep "^CPU(s):" | awk '{print $NF}') / 2)) LD_PRELOAD=libiomp5.so numactl -C $(seq -s, 0 2 $(($(lscpu | grep "^CPU(s):" | awk '{print $NF}') - 2))) -m 0 python demo.py -t /mnt/data/qwen-7b-chat -m /mnt/data/qwen-7b-chat-xft -d bf16 --chat true

root@xFT:/mnt/xFasterTransformer# cd /root/xFasterTransformer/examples/pytorch
OMP_NUM_THREADS=$(($(lscpu | grep "^CPU(s):" | awk '{print $NF}') / 2)) LD_PRELOAD=libiomp5.so numactl -C $(seq -s, 0 2 $(($(lscpu | grep "^CPU(s):" | awk '{print $NF}') - 2))) -m 0 python demo.py -t /mnt/data/qwen-7b-chat -m /mnt/da
ta/qwen-7b-chat-xft -d bf16 --chat true
ERROR: ld.so: object 'libiomp5.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object 'libiomp5.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
[INFO] xfastertransformer is installed, using pip installed package.
[INFO] SINGLE_INSTANCE MODE.

Please enter the prompt: Who are you?
==========================================================
I am a large-scale language model developed by Alibaba Cloud. My name is Qwen.
======================Performance=========================
Execution time:          xxx s
Latency:          xxx ms/token
Throughput:          xxx tokens/s

Please enter the prompt: Introduce Alibaba Cloud.
==========================================================
Alibaba Cloud is the cloud computing service provider of Alibaba Group. It offers a full range of cloud computing services, including computing, storage, databases, networking, security, big data, and artificial intelligence. Alibaba Cloud operates data centers around the world to deliver cloud computing services globally, supports a wide range of programming languages and development frameworks, and meets the varied requirements of enterprise customers.
======================Performance=========================
Execution time:          xxx s
Latency:          xxx ms/token
Throughput:          xxx tokens/s

Benchmark the model performance

The benchmark script uses dummy model data by default, so you do not need to prepare model data. Run the following command to test the model performance.

cd /root/xFasterTransformer/benchmark
XFT_CLOUD_ENV=1 bash run_benchmark.sh -m qwen-7b -d bf16 -bs 1 -in 32 -out 32 -i 10

Adjust the runtime parameters to test performance data for specific scenarios:

• -d or --dtype: Selects the model quantization type.

  • bf16 (default)

  • bf16_fp16

  • int8

  • bf16_int8

  • fp16

  • bf16_int4

  • int4

  • bf16_nf4

  • nf4

  • bf16_w8a8

  • w8a8

  • w8a8_int8

  • w8a8_int4

  • w8a8_nf4

• -bs or --batch_size: Selects the batch size. The default value is 1.

• -in or --input_tokens: Selects the input length. For a custom length, configure the corresponding prompt in prompt.json. The default value is 32.

• -out or --output_tokens: Selects the generation length. The default value is 32.

• -i or --iter: Selects the number of iterations. The larger the number of iterations, the longer the test takes to complete. The default value is 10.

The following is the execution result:

[Info] OMP_NUM_THREADS: 8
Run command line: mpirun    -n 1 bash run.sh 0 0 8 0
The folder '/data/models/Qwen-7B-Chat-xft' does not exist. Using fake weight!
[INFO] xfastertransformer is installed, using pip installed package.
[INFO] XFT_FAKE_MODEL is enabled. Using `export XFT_FAKE_LOAD_INFO=1` for more details.
=====start=======
[INFO] input argparse =  Namespace(batch_size=1, beam_width=1, chat=False, csv='', dtype='bf16', input_prompt=None, iteration=10, kv_cache_dtype='fp16', model_name='qwen-7b', model_path='/root/xFasterTransformer/benchmark/../examples/model_config/qwen-7b/', padding=False, prompt_path='/root/xFasterTransformer/benchmark/prompt.json', token_in='32', token_out=32, token_path='/root/xFasterTransformer/benchmark/../examples/model_config/qwen-7b/', warmup=2)
Once upon a time, there existed a little girl who liked to have adventures. She wanted to go to places and meet new people, and have fun.
Input token Length is 32
Batch_size: 1
Max_len: 64
======================================================
Start benchmark:
iteration 0 :
    Response: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
iteration 1 :
    Response: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
iteration 2 :
    Response: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
iteration 3 :
    Response: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
iteration 4 :
    Response: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
iteration 5 :
    Response: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
iteration 6 :
    Response: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
iteration 7 :
    Response: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
iteration 8 :
    Response: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
iteration 9 :
    Response: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

==============================qwen-7b Final Performance==============================
Inference Avg Latency:    xxx    ms
First token Avg Latency:      191.59 ms
Next token Max Latency:   xxx    ms
Next token Min Latency:   xxx    ms
Next token P90 Latency:   xxx    ms
Next token Avg Latency:   xxx    ms
Next token Latency:       xxx    ms
Throughput without 1st token:  5.95 tokens/s
================================================================================