How to deploy ChatGLM-6B on an AMD CPU instance-Elastic Compute Service(ECS)-阿里云帮助中心

This topic describes how to deploy the ChatGLM-6B language model on a g8a Alibaba Cloud AMD CPU instance.

Background

The ChatGLM-6B model is based on the General Language Model (GLM) architecture. It has 6.2 billion parameters, supports Chinese and English, and is optimized for Chinese question-answering and dialogue to generate responses that align with human preferences. Common use cases include self-cognition, outline writing, copywriting, email assistance, information extraction, role-playing, product comparison, and travel guidance.

Important

Alibaba Cloud does not guarantee the legality, security, or accuracy of the third-party model "ChatGLM-6B". Alibaba Cloud is not liable for any damages that may arise from the use of this model.
The ChatGLM-6B code is open-sourced under the Apache-2.0 license, and the use of its model weights is subject to the Model License. You must comply with the third-party model's user agreement, usage specifications, and all applicable laws and regulations. You are solely responsible for ensuring the legality and compliance of your use of the third-party model.
The ChatGLM-6B model weights are fully open for academic research. For free commercial use, you must submit a commercial license application. The model download shown in this topic is for demonstration purposes only. Alibaba Cloud is not liable for any damages that may arise from the use of this model.

Create an ECS instance

Go to the instance creation page.
Follow the on-screen instructions to configure the parameters and create an ECS instance.

Configure the following parameters. For information about other parameters, see Create an instance by using the custom launch template.
- Instance: ChatGLM-6B requires approximately 32 GiB of memory. To ensure stable model performance, select at least the ecs.g8a.4xlarge instance type, which provides 64 GiB of memory.
- Images: Alibaba Cloud Linux 3.2104 LTS 64-bit.
- Public IP Address: Select Assign Public IPv4 Address, set the bandwidth billing method to Pay-by-traffic, and set the peak bandwidth to 100 Mbps to accelerate model downloads.
- System Disk: The ChatGLM-6B model requires 26 GiB of storage space. To ensure the model runs smoothly, set the system disk size to 100 GiB.
Add a security group rule.

Add an inbound rule to the security group of your ECS instance to allow traffic on port 7860, which is used to access the WebUI service. For more information, see Add a security group rule.
After the instance is created, find its public IP address on the ECS Instances page.

Note
The public IP address is used to access the WebUI service for the AI chat.

In the instance list, find the address marked as "Public" in the IP Address column. This is the public IP address of the instance.

Deploy ChatGLM-6B

Manual

Software setup

Connect to the ECS instance.

For more information, see Connect to a Linux instance by using Workbench.

Install the software required to deploy ChatGLM-6B.

sudo yum install tmux git git-lfs wget curl gcc gcc-c++ tar unzip hwloc python38

Set Python 3.8 as the default version.

The system's default Python version (3.6) does not meet the minimum requirement for deploying ChatGLM-6B. You must switch to the installed Python 3.8.

sudo update-alternatives --config python

When prompted, enter 4 to select Python 3.8.

sudo update-alternatives --config python
There are 4 programs which provide 'python'.
  Selection    Command
-----------------------------------------------
*  1           /usr/libexec/no-python
+  2           /usr/bin/python3
   3           /usr/bin/python2
   4           /usr/bin/python3.8
Enter to keep the current selection[+], or type selection number: 4

Upgrade the pip version for Python.

sudo python -m ensurepip --upgrade
sudo python -m pip install --upgrade pip

Enable Git LFS.

Git LFS is required to download the pre-trained model.
```
git lfs install
```

Download source code and model

Create a tmux session.
```
tmux
```
Important
Downloading the pre-trained model can take a long time, and success depends on network conditions. We recommend running the download in a tmux session to prevent interruptions from a lost remote connection.

Download the ChatGLM-6B project source code and the chatglm-6b pre-trained model.

git clone https://github.com/THUDM/ChatGLM-6B.git
git clone https://huggingface.co/THUDM/chatglm-6b

Note

If the chatglm-6b model download fails, run the following commands to download the pre-trained model from ModelScope.

git clone https://www.modelscope.cn/ZhipuAI/ChatGLM-6B.git chatglm-6b
git -C chatglm-6b checkout v1.0.16
sed -i '33c from .configuration_chatglm import ChatGLMConfig' chatglm-6b/modeling_chatglm.py

View the contents of the current directory.

ls -l

After the download is complete, the current directory contains the following:

[test@iZ2ze1a6e1z    ~]$ ls -l
total 8
drwxrwxr-x  3 test test 4096 Aug  4 11:38 chatglm-6b
drwxrwxr-x  9 test test 4096 Aug  4 11:12 ChatGLM-6B

Runtime environment

Change the pip download source.

We recommend changing the pip download source to accelerate dependency installation.
1. Create a pip configuration directory.
```
mkdir -p ~/.config/pip
```
2. Configure the pip installation mirror.
```
cat > ~/.config/pip/pip.conf <<EOF
[global]
index-url=http://mirrors.cloud.aliyuncs.com/pypi/simple/
[install]
trusted-host=mirrors.cloud.aliyuncs.com
EOF
```

Install dependencies required by ZenDNN.

ZenDNN requires the CPU version of PyTorch, so you must manually install its dependencies.

pip install protobuf transformers==4.27.1 cpm_kernels gradio mdtex2html sentencepiece accelerate streamlit streamlit-chat
pip install torch==1.12.0+cpu torchvision==0.13.0+cpu --extra-index-url https://download.pytorch.org/whl/cpu

Download and install ZenDNN.

The ZenDNN runtime library provides APIs with basic neural network building blocks optimized for the AMD CPU architecture. It enables developers of deep learning applications and frameworks to improve inference performance on AMD CPUs.

wget https://download.amd.com/developer/eula/zendnn/zendnn-4-0/pytorch-end-user-license/PT_v1.12_ZenDNN_v4.0_Python_v3.8.zip
unzip PT_v1.12_ZenDNN_v4.0_Python_v3.8.zip
cd PT_v1.12_ZenDNN_v4.0_Python_v3.8/
source scripts/PT_ZenDNN_setup_release.sh

The following message indicates that ZenDNN is installed successfully.

ZENDNN_PARENT_FOLDER: /home/test/PT_v1.12_ZenDNN_v4.0_Python_v3.8
./torch-1.12-cp38-cp38-manylinux2014_x86_64.whl
install PT whl
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: http://mirrors.cloud.aliyuncs.com/pypi/simple/
Processing ./torch-1.12-cp38-cp38-manylinux2014_x86_64.whl
Requirement already satisfied: typing-extensions in /home/test/.local/lib/python3.8/site-packages (from torch==1.12) (4.7.1)
Installing collected packages: torch
  Attempting uninstall: torch
    Found existing installation: torch 1.12.0+cpu
    Uninstalling torch-1.12.0+cpu:
      Successfully uninstalled torch-1.12.0+cpu
Successfully installed torch-1.12
To check the installed version of PT:
1.12
ZenDNN Release Setup is complete
-------------------------------

Set the OMP_NUM_THREADS and GOMP_CPU_AFFINITY environment variables.

The ZenDNN runtime library requires you to explicitly set the OMP_NUM_THREADS and GOMP_CPU_AFFINITY environment variables for the hardware platform.
```
sudo bash -c 'cat > /etc/profile.d/env.sh' << EOF
export OMP_NUM_THREADS=\$(nproc --all)
export GOMP_CPU_AFFINITY=0-\$(( \$(nproc --all) - 1 ))
EOF
source /etc/profile
```

AI chat

Run the following commands to start the WebUI service.

cd ChatGLM-6B
sed -i 's?THUDM/chatglm-6b?../chatglm-6b?g' web_demo2.py
sed -i 's/.half().cuda()/.float()/g' web_demo2.py
streamlit run web_demo2.py --server.port 7860

When the following message appears, the WebUI service has started successfully.

Collecting usage statistics. To deactivate, set browser.gatherUsageStats to False.
  You can now view your Streamlit app in your browser.
  Network URL: http://172.17.xxx:7860
  External URL: http://123.57.xxx:7860

In your browser's address bar, enter http://<ecs_instance_public_ip>:7860 to access the web UI.
In the Input box, enter a prompt and click Send to start the AI chat.

The left panel lets you adjust the inference parameters. You can use the sliders to set max_length (maximum generation length, range: 0–4096), top_p (sampling probability, range: 0.00–1.00), and temperature (randomness, range: 0.00–1.00).

Automatic

Alibaba Cloud provides an automatic deployment script to deploy and run the ChatGLM-6B model with a single command.

Log in to the ECS instance as the root user.

For more information, see Connect to a Linux instance by using Workbench.
Install tmux and create a tmux session.
```
yum install -y tmux
tmux
```
Note
The script downloads model files, which can take a long time. We recommend running the deployment in a tmux session to prevent interruptions from a lost remote connection.

Download the automatic deployment script.

wget https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20230920/afjx/deploy_chatglm-6b.sh

Make the automatic deployment script executable.
```
chmod +x deploy_chatglm-6b.sh
```
Run the automatic deployment script.
```
./deploy_chatglm-6b.sh
```
The following output indicates that the automatic deployment is complete.
```
[root@iZ2ze5ec5xxx ~]# chmod +x deploy_chatglm-6b.sh
nohup: ignoring input
MAKRER=SHOW_LOCALE;printf $MAKRER
Collecting usage statistics. To deactivate, set browser.gatherUsageStats to False.
You can now view your Streamlit app in your browser.
Network URL: http://172.17.xxx.xxx:7860
External URL: http://39.107.xxx.xxx:7860
```
Note
This script takes a long time to run. If your connection to the ECS instance is lost during deployment in a tmux session, log in to the instance again and run the tmux attach command to restore the session and check the script's progress.
In your browser's address bar, enter http://<ecs_instance_public_ip>:7860 to access the web UI.
In the Input box, enter a prompt and click Send to start the AI chat.