Build a knowledge base application using a large language model and vector retrieval on ACK

• Add GPU nodes to the cluster. The GPU instance type must be from the V100 or A10 series. vGPUs are not supported. To prevent out-of-memory errors, use a GPU instance type with at least 24 GiB of GPU memory.

• Configure a NAT Gateway for your cluster to provide public network access to the cluster's nodes and applications.

How do I change the model?

1. Download the model.

   To deploy other open-source models, download them from Hugging Face or ModelScope and store them in Object Storage Service (OSS) or Network Attached Storage (NAS). The following table lists some example models. For a complete list of compatible types, see Langchain-Chatchat.

   Model type	Model name	Container model path
   DeepGPU-LLM converted model	qwen-7b-chat-aiacc	/llm-model/qwen-7b-chat-aiacc
   DeepGPU-LLM converted model	qwen-14b-chat-aiacc	/llm-model/qwen-14b-chat-aiacc
   DeepGPU-LLM converted model	chatglm2-6b-aiacc	/llm-model/chatglm2-6b-aiacc
   DeepGPU-LLM converted model	baichuan2-7b-chat-aiacc	/llm-model/baichuan2-7b-chat-aiacc
   DeepGPU-LLM converted model	baichuan2-13b-chat-aiacc	/llm-model/baichuan2-13b-chat-aiacc
   DeepGPU-LLM converted model	llama-2-7b-hf-aiacc	/llm-model/llama-2-7b-hf-aiacc
   DeepGPU-LLM converted model	llama-2-13b-hf-aiacc	/llm-model/llama-2-13b-hf-aiacc
   Open-source model	qwen-7b-chat	/llm-model/Qwen-7B-Chat
   Open-source model	qwen-14b-chat	/llm-model/Qwen-14B-Chat
   Open-source model	chatglm2-6b	/llm-model/chatglm2-6b
   Open-source model	chatglm2-6b-32k	/llm-model/chatglm2-6b-32k
   Open-source model	baichuan2-7b-chat	/llm-model/Baichuan2-7B-Chat
   Open-source model	baichuan2-13b-chat	/llm-model/Baichuan2-13B-Chat
   Open-source model	llama-2-7b-hf	/llm-model/Llama-2-7b-hf
   Open-source model	llama-2-13b-hf	/llm-model/Llama-2-13b-hf

2. (Optional) Convert the model.

   The Langchain-Chatchat project in this topic includes the DeepGPU-LLM model converter and uses the DeepGPU-LLM accelerated model qwen-7b-chat-aiacc by default.

   To use DeepGPU-LLM for inference optimization on other open-source LLMs, you must first convert the Hugging Face models to the format that DeepGPU-LLM supports.

   For example, to convert qwen-7b-chat, run the following command in the container to convert the model format. Alternatively, you can install the DeepGPU-LLM inference engine and use it to convert the model format. For more information, see Install and use DeepGPU-LLM.

   # qwen-7b weight convert
   huggingface_qwen_convert \
       -in_file /llm-model/Qwen-7B-Chat \
       -saved_dir /llm-model/qwen-7b-chat-aiacc \
       -infer_gpu_num 1 \
       -weight_data_type fp16 \
       -model_name qwen-7b

3. Create a static PersistentVolume and PersistentVolumeClaim.

   1. This example uses OSS. After the model is downloaded, run the following command to create a Secret.

      kubectl create -f oss-secret.yaml

      The following is an example oss-secret.yaml file for creating a Secret. Replace <your AccessKeyID> and <your AccessKeySecret> with your credentials.

      apiVersion: v1
      kind: Secret
      metadata:
        name: oss-secret
        namespace: default
      stringData:
        akId: <your AccessKeyID>
        akSecret: <your AccessKeySecret>

   2. Run the following command to create a static PersistentVolume.

      kubectl create -f model-oss.yaml

      The following is an example model-oss.yaml file for creating a static PersistentVolume. Replace "<Your_Bucket_Name>" and "<Your_OSS_Endpoint>" with your bucket name and endpoint URL.

      apiVersion: v1
      kind: PersistentVolume
      metadata:
        name: model-oss
        labels:
          alicloud-pvname: model-oss
      spec:
        capacity:
          storage: 30Gi 
        accessModes:
          - ReadOnlyMany
        persistentVolumeReclaimPolicy: Retain
        csi:
          driver: ossplugin.csi.alibabacloud.com
          volumeHandle: model-oss
          nodePublishSecretRef:
            name: oss-secret
            namespace: default
          volumeAttributes:
            bucket: "<Your_Bucket_Name>"
            url: "<Your_OSS_Endpoint>" # The URL in this topic is oss-cn-hangzhou.aliyuncs.com.
            otherOpts: "-o umask=022 -o max_stat_cache_size=0 -o allow_other"
            path: "/"

   3. Run the following command to create a static PersistentVolumeClaim.

      kubectl create -f model-pvc.yaml

      The following is an example model-pvc.yaml file for creating a static PersistentVolumeClaim.

      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: model-pvc
      spec:
        accessModes:
          - ReadOnlyMany
        resources:
          requests:
            storage: 30Gi
        selector:
          matchLabels:
            alicloud-pvname: model-oss

   For specific parameter configurations, see Use a static ossfs 1.0 volume.

4. Update the Helm values.

   1. Log on to the ACK console. In the left navigation pane, click Clusters.

   2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Applications > Helm.

   3. On the Helm page, find the deployed langchain-chatchat service and click Update in the Actions column. Change llm.model to the new model name and llm.modelPVC to the name of the PVC that stores the new model. For the model name and model mount path, see List of supported models.

How do I deploy using ECI Pods?

Set the llm.pod.instanceType and chat.pod.instanceType parameters to eci. The default Annotation and Label configurations for the ECI type are as follows. For more information about Annotations, see ECI Pod Annotation.

annotations:
  k8s.aliyun.com/eci-use-specs: ecs.gn6i-c8g1.2xlarge,ecs.gn6i-c16g1.4xlarge,ecs.gn6v-c8g1.8xlarge,ecs.gn7i-c8g1.2xlarge,ecs.gn7i-c16g1.4xlarge
  k8s.aliyun.com/eci-extra-ephemeral-storage: "50Gi"
labels:
  alibabacloud.com/eci: "true"

If you change the image or model, you must modify the k8s.aliyun.com/eci-use-specs and k8s.aliyun.com/eci-extra-ephemeral-storage annotations. Otherwise, the application will fail to start due to insufficient resources. For more information about ECI billing, see Billing overview.

How do I accelerate text vectorization?

The application's built-in embedding model is text2vec-bge-large-chinese. For more information, see text2vec-bge-large-chinese.

By default, the chat application runs the embedding model on the CPU, but you can accelerate text vectorization by requesting GPU resources in chat.pod.resources.

...
chat:
  pod:
    resources:
      limits:
        nvidia.com/gpu: "1"
      requests:
        cpu: "4"
        memory: 8Gi
        nvidia.com/gpu: "1"

How do I specify the vector database type?

Supported vector databases include Faiss and Alibaba Cloud AnalyticDB for PostgreSQL.

• Faiss is an open-source, in-memory vector library developed by Facebook. Faiss is deployed in the chat Pod and is therefore constrained by the Pod's resources. If you use Faiss, we recommend increasing the chat Pod's memory.

• AnalyticDB for PostgreSQL is a massively parallel processing (MPP) data warehousing service that provides online analysis of massive datasets.

To use AnalyticDB for PostgreSQL with this project, your instance must meet the following conditions:

• The vector engine optimization feature must be enabled.

• The compute node specification must be 4-core 16 GiB or higher.

• Modify the parameter db.dbType to adb, and enter the database information in db.adb.

db:
  dbType: adb
  embeddingModel: text2vec-bge-large-chinese
  adb:
    pgHost: "pg.host.demo.com"
    pgPort: "5432"
    pgDataBase: "langchain"
    pgUser: "langchain"
    pgPassword: "pg_password"

How do I change the number of service replicas?

Set llm.pod.replicas to the desired number of inference service replicas.

llm:
  model: qwen-7b-chat-aiacc
  modelPVC: "" # PVC name.
  load8bit: true
  pod:
    replicas: 1 # The number of replicas for the model inference service.

Set chat.pod.replicas to the desired number of application service replicas.

chat:
 kbPVC: ""
 pod:
   replicas: 1 # The number of replicas for the model application service.