在ACK中使用KTransformers部署DeepSeek-R1模型

执行以下命令，基于KTransformers推理框架部署DeepSeek-R1-Q4_K_M量化模型的推理服务。

arena serve custom \
--name=ktransformers \
--version=v1 \
--gpus=4 \
--replicas=1 \
--restful-port=8080 \
--image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:v0.7.2-ktransformer \
--share-memory=30Gi \
--data="llm-model:/models/DeepSeek-R1-GGUF" \
"cd /vllm-workspace/ktransformers/;PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python3 ./ktransformers/server/main.py --gguf_path /models/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M/ --model_path /models/DeepSeek-R1-GGUF/ --model_name unsloth/DeepSeek-R1-Q4_K_M --cpu_infer 65 --max_new_tokens 8192 --cache_lens 32768 --total_context 32768 --cache_q4 true --temperature 0.6 --top_p 0.95 --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --force_think --use_cuda_graph --host 0.0.0.0 --port 8080"

预期输出：

service/ktransformers-v1 created
deployment.apps/ktransformers-v1-custom-serving created
INFO[0004] The Job ktransformers has been submitted successfully
INFO[0004] You can run `arena serve get ktransformers --type custom-serving -n default` to check the job status

输出结果表明已经成功部署了一个与vLLM相关的服务及其配套的Deployment和Job。

执行以下命令，查看推理服务的详细信息。

arena serve get ktransformers

预期输出：

$ arena serve get ktransformers
# ================================
Name:       ktransformers
Namespace:  default
Type:       Custom
Version:    v1
Desired:    1
Available:  1
Age:        2m
Address:    192.168.XXX.XXX
Port:       RESTFUL:8080
GPU:        4

Instances:
  NAME                                              STATUS   AGE  READY  RESTARTS  GPU  NODE
  ----                                              ------   ---  -----  --------  ---  ----
  ktransformers-v1-custom-serving-XXXb549799-XXXbj  Running  2m   1/1    0         4    XX-XX-1.10.6.17.38

使用以下YAML，创建OpenWebUI应用。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: openwebui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: openwebui
  template:
    metadata:
      labels:
        app: openwebui
    spec:
      containers:
      - env:
        - name: ENABLE_OPENAI_API
          value: "True"
        - name: ENABLE_OLLAMA_API
          value: "False"
        - name: OPENAI_API_BASE_URL
          value: http://ktransformers-v1:8080/v1
        - name: ENABLE_AUTOCOMPLETE_GENERATION
          value: "False"
        - name: ENABLE_TAGS_GENERATION
          value: "False"
        image: kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/open-webui:main
        name: openwebui
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - mountPath: /app/backend/data
          name: data-volume
      volumes:
      - emptyDir: {}
        name: data-volume
---
apiVersion: v1
kind: Service
metadata:
  name: openwebui
  labels:
    app: openwebui
spec:
  type: ClusterIP
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: openwebui

使用kubectl port-forward在推理服务与本地环境间建立端口转发。

kubectl port-forward svc/openwebui 8080:8080

预期输出：

Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

访问http://localhost:8080，登录OpenWebUI页面。

第一次登录需要配置管理员账号和密码。输入提示词，预期输出如下图所示。