Accelerate the startup of Large Language Model inference service pods in Knative services using Fluid-Container Service for Kubernetes(ACK)-阿里云帮助中心

Fluid is an open source, Kubernetes-native distributed dataset orchestration and acceleration engine. It is mainly used for data-intensive applications in cloud-native scenarios, such as big data and AI applications. This topic describes how to use Fluid in Knative to accelerate the startup of model inference service pods and improve application response efficiency.

The example in this topic uses the Qwen-7B-Chat-Int8 model and the ecs.gn6v-c8g1.2xlarge GPU instance type. Monitoring shows that the time from pod startup until the model is ready is approximately 30 seconds with Fluid enabled and 60 seconds without it. Using Fluid can reduce the model ready time by almost half, which significantly improves service response efficiency.

Important

The data in this example is for reference only. Actual results may vary based on your operating environment.

The procedure is as follows.

Complete the prerequisites. Then, download the sample data model and upload it to an OSS bucket.
Accelerate model loading using Fluid. This step includes configuring a dataset, using JindoRuntime to accelerate dataset access, and preloading the dataset with a DataLoad task.
Deploy and access the inference service to evaluate the improvement in startup efficiency provided by Fluid.

Prerequisites

You have deployed Knative in a cluster that runs version 1.22 or later. For more information, see Deploy and manage Knative components.
You have created a node pool that contains GPU-accelerated instances. The instance type must be V100, A10, or T4, and the CUDA version must be 12.0 or later. For more information, see Add GPU nodes to a cluster.
The GPU nodes must use driver version 525. To specify driver version 525.105.17, add the ack.aliyun.com/nvidia-driver-version:525.105.17 label to the GPU node pool. For more information, see Customize the GPU driver version for a node by specifying a version number.
You have created an OSS bucket or an NAS file system to store model data.
This topic uses an OSS bucket as an example. For more information, see Activate OSS and Create buckets. For information about related fees, see Billing overview.
You have installed the cloud-native AI suite and deployed the ack-fluid component. For more information, see Install the cloud-native AI suite.

Step 1: Prepare the model data and upload it to an OSS bucket

You can use OSS or NAS to prepare the model data. This topic uses OSS as an example.

Run the following command to install Git.
```
sudo yum install git
```
Run the following command to install the Git Large File Storage (LFS) plugin.
```
sudo yum install git-lfs
```
Run the following command to clone the Qwen-7B-Chat-Int8 repository from ModelScope to your local machine.
```
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/qwen/Qwen-7B-Chat-Int8.git
```
Run the following command to navigate to the Qwen-7B-Chat-Int8 repository folder.
```
cd Qwen-7B-Chat-Int8
```
Run the following command in the Qwen-7B-Chat-Int8 folder to download the large files managed by Git LFS.
```
git lfs pull
```
Upload the downloaded Qwen-7B-Chat-Int8 files to an OSS bucket. For more information, see Simple upload.
You can use the ossutil command-line interface to upload files. For more information about how to install and configure ossutil, see Install ossutil.
Run the following command to create a folder named Qwen-7B-Chat-Int8 in OSS.
```
ossutil mkdir oss://<Your-Bucket-Name>/Qwen-7B-Chat-Int8
```

Run the following command to upload the model files to OSS.

ossutil cp -r ./Qwen-7B-Chat-Int8 oss://<Your-Bucket-Name>/Qwen-7B-Chat-Int8

Configure a persistent volume (PV) and a persistent volume claim (PVC) named llm-model for the target cluster. For more information, see Use a static ossfs 1.0 persistent volume.

The following example shows a PV configuration.

Configuration item	Description
Volume type	OSS.
Name	llm-model.
Access credential	Configure the AccessKey ID and AccessKey secret to access OSS.
Bucket ID	Select the OSS bucket that you created.
OSS path	Select the path where the model is stored, such as /Qwen-7B-Chat-Int8.

The following is an example PVC configuration.

Configuration item	Description
PVC type	OSS.
Name	llm-model.
Allocation mode	Select Existing PV.
Existing Persistent Volume (PV)	Click the Select Existing PV link and select the PV that you created.

Step 2: Accelerate model loading using Fluid

To improve the startup efficiency of application pods, you can create and prepare the Jindo FUSE pod in advance when you use Fluid to accelerate data access to the model PV. This prevents the application pod from waiting for the Jindo FUSE pod to become ready during startup. This approach reduces latency and ensures that the service can respond to user requests more quickly.

Use the following sample code to configure a dataset and a JindoRuntime to accelerate dataset access.

apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
  name: qwen-7b-chat-int8-dataset
spec:
  mounts:
    - mountPoint: pvc://llm-model # The name of the PVC that you created.
      name: data
      path: /
  accessModes:
    - ReadOnlyMany
---
apiVersion: data.fluid.io/v1alpha1
kind: JindoRuntime
metadata:
  name: qwen-7b-chat-int8-dataset
spec:
  replicas: 2
  tieredstore:
    levels:
      # Set the storage medium to memory.
      - mediumtype: MEM
        volumeType: emptyDir
        path: /dev/shm
        quota: 20Gi
        high: "0.9"
        low: "0.8"

Expected output:

$ kubectl get datasets.data.fluid.io
NAME                        UFS TOTAL SIZE   CACHED   CACHE CAPACITY   CACHED PERCENTAGE   PHASE   AGE
qwen-7b-chat-int8-dataset   17.01GiB         0.00B    20.00GiB         0.0%                Bound   89s
$ kubectl get pvc
NAME                        STATUS   VOLUME                              CAPACITY   ACCESS MODES   STORAGECLASS   AGE
qwen-7b-chat-int8-dataset   Bound    default-qwen-7b-chat-int8-dataset   100Pi      ROX            fluid          53s

Use the following sample code to create a DataLoad task that preloads the dataset you created in the previous step.

apiVersion: data.fluid.io/v1alpha1
kind: DataLoad
metadata:
  name: dataset-warmup
spec:
  dataset:
    name: qwen-7b-chat-int8-dataset
    namespace: default
  loadMetadata: true
  target:
    - path: /
      replicas: 1

Expected output:

$ kubectl get dataloads
NAME             DATASET                     PHASE      AGE     DURATION
dataset-warmup   qwen-7b-chat-int8-dataset   Complete   2m27s   1m46s
$ kubectl get dataset
NAME                        UFS TOTAL SIZE   CACHED     CACHE CAPACITY   CACHED PERCENTAGE   PHASE   AGE
qwen-7b-chat-int8-dataset   17.01GiB         17.01GiB   20.00GiB         100.0%              Bound   10m

Step 3: Deploy and access the inference service

Use the following code to deploy the model inference service.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  labels:
    release: qwen-test
  name: qwen-test
  namespace: default
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/metric: "concurrency" # Set concurrency as the metric for auto scaling.
        autoscaling.knative.dev/target: "2" # Set the target number of concurrent requests to 2.
        autoscaling.knative.dev/max-scale: "3" 
      labels:
        release: qwen-test
    spec:
      containers:
      - command:
        - sh
        - -c
        - python3 -m vllm.entrypoints.openai.api_server --port 8080 --trust-remote-code
          --served-model-name qwen --model /mnt/models/Qwen-7B-Chat-Int8 --gpu-memory-utilization
          0.95 --quantization gptq --max-model-len=6144
        image: kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:0.4.1
        imagePullPolicy: IfNotPresent
        name: vllm-container
        readinessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
        resources:
          limits:
            cpu: "32"
            memory: 64Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: "4"
            memory: 8Gi
            nvidia.com/gpu: "1"
        volumeMounts:
        - mountPath: /mnt/models/Qwen-7B-Chat-Int8
          name: qwen-7b-chat-int8
      volumes:
      - name: qwen-7b-chat-int8
        persistentVolumeClaim:
          claimName: qwen-7b-chat-int8-dataset # The PVC created from the dataset.

Expected output:

$kubectl get ksvc
NAME                             URL                                                         LATESTCREATED                          LATESTREADY                            READY   REASON
qwen-test                        http://qwen-test.default.example.com                        qwen-test-00001                        qwen-test-00001                        True

Access the created service.

curl -H "Host: qwen-test.default.example.com" -H "Content-Type: application/json" http://1XX.XX.XX.XXX:80/v1/chat/completions -d '{"model": "qwen", "messages": [{"role": "user", "content": "Hangzhou West Lake"}], "max_tokens": 10, "temperature": 0.7, "top_p": 0.9, "seed": 10}'

Expected output:

{"id":"cmpl-c496a518436e4e99bfb36617ac56a25d","object":"chat.completion","created":1736740430,"model":"qwen","choices":[{"index":0,"message":{"role":"assistant","content":"West Lake is located in Hangzhou, Zhejiang Province, China. It is a famous scenic spot"},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":10,"total_tokens":20,"completion_tokens":10}}

The time from pod startup to model readiness is now about 30 seconds, a significant improvement in service response efficiency compared to deployments without Fluid.

References

Deploy vLLM inference applications using Knative: Use Knative Pod Autoscaler (KPA) to adjust resource allocation based on queries per second (QPS) or records per second (RPS) to improve inference service performance.
Best practices for deploying AI model inference services in Knative: Improve AI inference service capabilities and GPU resource utilization by accelerating model deployment, enabling auto scaling, and using shared GPU scheduling.