Deploy ChatGLM2-6B on Intel c8i instances

Background information

Create an ECS instance

1. Go to the instance creation page.

2. Follow the on-screen instructions to configure the parameters and create an ECS instance.

   Pay special attention to the following parameters. For information about other parameters, see Create an instance by using the wizard.

   • Instance: ChatGLM2-6B requires approximately 16 GiB of memory. To ensure stable operation, select an instance type of at least ecs.c8i.4xlarge (32 GiB of memory).

   • Images: Alibaba Cloud Linux 3.2104 LTS 64-bit.

   • Public IP Address: Select Assign Public IPv4 Address. For the bandwidth billing method, select Pay-by-traffic and set the peak bandwidth to 100 Mbps. This helps speed up model downloads.

   • System Disk: The ChatGLM2-6B model requires 38 GiB of storage space for downloading, converting, and running. To ensure the model runs smoothly, we recommend setting the system disk size to 100 GiB.

3. Add a security group rule.

   Add an inbound rule to the instance's security group to allow traffic on port 22 for SSH and port 7860 for the Web UI. For more information, see Add a security group rule.

Install the container environment

1. Remotely connect to the ECS instance.

   For more information, see Connect to a Linux instance by using Workbench.

2. Install and start Docker.

   For more information, see Install Docker on an Alibaba Cloud Linux 3 instance.

3. Pull and run the Intel xFasterTransformer container.

   sudo docker pull intel/xfastertransformer:1.3.1
   sudo docker run -it --name xFT -h xFT --privileged --shm-size=16g --network host -v /mnt:/mnt -w /mnt/xFasterTransformer intel/xfastertransformer:1.3.1

   The following output indicates that the container was pulled and is running successfully.

   [ecs-user@iZ2zeb16jjz xxx ~]$ sudo docker pull intel/xfastertransformer:1.3.1
   1.3.1: Pulling from intel/xfastertransformer
   df2fac849a45: Pull complete
   3bb6bedb0d39: Pull complete
   835b24f77466: Pull complete
   6b0e454047b1: Pull complete
   9e44a44e2c9e: Pull complete
   1bf21f422f88: Pull complete
   afd79455bacf: Pull complete
   43108065c9e6: Pull complete
   6afbdced9f70: Pull complete
   af1f8308af03: Pull complete
   315d558533af: Pull complete
   15d21a253a14: Pull complete
   b4a79fb1620e: Pull complete
   9e4b3e35e158: Pull complete
   afb69a4ac294: Pull complete
   3c62864266ba: Pull complete
   923b0f5c2911: Pull complete
   913a43a29dc5: Pull complete
   e239b5ababe1: Pull complete
   Digest: sha256:ba47ef632d26055255a9b75e614fb92dbfa6874355e1f2axxx
   Status: Downloaded newer image for intel/xfastertransformer:1.3.1
   docker.io/intel/xfastertransformer:1.3.1
   [ecs-user@iZ2zeb16jjzaoc xxx ~]$ sudo docker run -it --name xFT -h xFT --privileged --shm-size=16g --network host -v /mnt:/mnt -w /mnt/xFasterTransformer intel/xfastertransformer:1.3.1

   Important

   All subsequent steps must be performed inside the container. If you exit the container, run the following commands to start it and enter its shell environment again.

   sudo docker start xFT
   sudo docker exec -it xFT bash

4. (Optional) Update the xFasterTransformer scripts.

   The xFasterTransformer image includes the corresponding scripts. You can update to the latest test scripts if needed.

   apt update
   apt install -y git
   cd /root/xFasterTransformer
   git pull

Prepare the model data

1. Install dependencies in the container.

   apt update
   apt install -y wget git git-lfs vim tmux

2. Enable Git LFS.

   Git LFS is required to download the pre-trained model.

   git lfs install

3. Create and go to the model data directory.

   mkdir /mnt/data
   cd /mnt/data

4. Create a tmux session.

   tmux

   Important

   Downloading the pre-trained model can take a long time, and success depends on your network conditions. We recommend running the download in a tmux session to prevent the process from being interrupted if your connection to the ECS instance is lost.

5. Download the ChatGLM2-6B pre-trained model.

   git clone --depth 1 https://www.modelscope.cn/ZhipuAI/chatglm2-6b.git /mnt/data/chatglm2-6b

6. Convert the model data.

   The downloaded model is in Hugging Face format and must be converted to the xFasterTransformer format. The converted model is saved to the /mnt/data/chatglm2-6b-xft folder.

   python -c 'import xfastertransformer as xft; xft.ChatGLM2Convert().convert("/mnt/data/chatglm2-6b")'

   Note

   Different models require different conversion classes. xFasterTransformer supports the following model conversion classes:

   • LlamaConvert

   • ChatGLMConvert

   • ChatGLM2Convert

   • ChatGLM3Convert

   • OPTConvert

   • BaichuanConvert

   • QwenConvert

Run the model

cd /root/xFasterTransformer/examples/pytorch
OMP_NUM_THREADS=$(($(lscpu | grep "^CPU(s):" | awk '{print $NF}') / 2)) LD_PRELOAD=libiomp5.so numactl -C $(seq -s, 0 2 $(($(lscpu | grep "^CPU(s):" | awk '{print $NF}') - 2))) -m 0 python demo.py -t /mnt/data/chatglm2-6b -m /mnt/data/chatglm2-6b-xft -d bf16 --chat true

Please enter the prompt: Please introduce Alibaba Cloud.
==========================================
Alibaba Cloud is the cloud computing arm of Alibaba Group. It provides a wide range of cloud products and services to help enterprises and government organizations achieve digital transformation and cloud adoption. Here is a brief introduction:
1. Service scope: Alibaba Cloud provides cloud products and services such as compute, storage, databases, networking, security, and AI. These can meet the cloud needs of enterprises and government organizations with different scales and requirements.
2. Technology leadership: Alibaba Cloud has many leading cloud technologies, such as the Apsara operating system.
====================Performance====================
Execution time: xxx s
Latency:        xxx ms/token
Throughput:     xxx tokens/s
Please enter the prompt:

Benchmark model performance

The benchmark script uses fake model data by default, so you do not need to prepare model data. Run the following command to test the model's performance.

cd /root/xFasterTransformer/benchmark
XFT_CLOUD_ENV=1 bash run_benchmark.sh -m chatglm2-6b -d bf16 -bs 1 -in 32 -out 32 -i 10

Adjust the runtime parameters to test performance in specific scenarios:

• Use -d or --dtype to select the model quantization type:

  • bf16 (default)

  • bf16_fp16

  • int8

  • bf16_int8

  • fp16

  • bf16_int4

  • int4

  • bf16_nf4

  • nf4

  • bf16_w8a8

  • w8a8

  • w8a8_int8

  • w8a8_int4

  • w8a8_nf4

• Use -bs or --batch_size to select the batch size. The default value is 1.

• Use -in or --input_tokens to select the input length. For custom lengths, configure the corresponding prompt in prompt.json. The default value is 32.

• Use -out or --output_tokens to select the generation length. The default value is 32.

• Use -i or --iter to set the number of iterations. More iterations result in a longer wait time for test results. The default value is 10.

The following is sample output:

root@xFT:~/xFasterTransformer# cd /root/xFasterTransformer/benchmark
root@xFT:~/xFasterTransformer/benchmark# XFT_CLOUD_ENV=1 bash run_benchmark.sh -m chatglm2-6b -d bf16 -bs 1 -in 32 -out 32 -i 10
You are using model chatglm2-6b, dtype bf16, batch size 1, input tokens 32, output tokens 32, beam width 1 and iteration 10 on 1 sockets system.
General Test mode:
        The mapping method for CPU IDs in the cloud server environment is different,
        for example, (0,1), (2,3), (...) where consecutive pairs of CPU IDs belong
        to a single physical core. In this mapping relationship,
        you can enable `export XFT_CLOUD_ENV=1` to bind to the correct physical core.
OMP_NUM_THREADS: 8
Run command line: numactl -C 0,2,4,6,8,10,12,14 -m 0 python /root/xFasterTransformer/benchmark/benchmark.py     --token_path /root/xFasterTransformer/benchmark/../examples/model_config/chatglm2-6b/     --model_path /root/xFasterTransformer/benchmark/../examples/model_config/chatglm2-6b/     --prompt_path /root/xFasterTransformer/benchmark/prompt.json     --model_name chatglm2-6b     --dtype bf16     --batch_size 1     --token_in 32     --token_out 32
    --beam_width 1    --iteration 10    --warmup 2
[INFO] xfastertransformer is installed, using pip installed package.
[INFO] XFT_FAKE_MODEL is enabled. Using `export XFT_FAKE_LOAD_INFO=1` for more details.
[INFO] SINGLE_INSTANCE MODE.
======start=======
In ancient times, the goddess Nüwa mended the heavens with smelted stones, using all but one of the 36,501 stones she had prepared.
Input token Length is 32
Batch_size: 1
Max_len: 64
==========================================================
Start benchmark:
iteration 0 :
iteration 1 :
iteration 2 :
iteration 3 :
iteration 4 :
iteration 5 :
iteration 6 :
iteration 7 :
iteration 8 :
iteration 9 :
==========================================chatglm2-6b Final Performance==============================================
Inference Avg Latency:    xxx ms
First token Avg Latency:       xxx ms
Next token Max Latency:   xxx ms
Next token Min Latency:   xxx ms
Next token P90 Latency:   xxx ms
Next token Avg Latency:   xxx ms
Next token Latency:       xxx ms
Throughput without 1st token:  xxx tokens