Configure GPU resources and enable GPU sharing for Knative

Prerequisites

• Knative is deployed in your cluster.

Configure GPU resources

Add the k8s.aliyun.com/eci-use-specs annotation to spec.template.metadata.annotations to specify a GPU-accelerated Elastic Compute Service (ECS) instance type. Add the nvidia.com/gpu field to spec.containers.resources.limits to set the GPU count. If you omit nvidia.com/gpu, the pod reports errors after it starts.

Example Knative Service with one GPU:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-go
spec:
  template:
    metadata:
      labels:
        app: helloworld-go
      annotations:
        k8s.aliyun.com/eci-use-specs: ecs.gn5i-c4g1.xlarge  # GPU-accelerated ECS instance type
    spec:
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/knative-sample/helloworld-go:73fbdd56
          ports:
          - containerPort: 8080
          resources:
            limits:
              nvidia.com/gpu: '1'    # Number of GPUs required. Required field.

Enable GPU sharing

GPU sharing lets multiple pods share a physical GPU, reducing costs when workloads don't need dedicated GPU access.

To enable GPU sharing:

1. Enable GPU sharing for nodes.

2. Add the aliyun.com/gpu-mem field to spec.containers.resources.limits in your Knative Service to set the GPU memory size:

   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
     name: helloworld-go
     namespace: default
   spec:
     template:
       metadata:
         annotations:
           autoscaling.knative.dev/maxScale: "100"
           autoscaling.knative.dev/minScale: "0"
       spec:
         containerConcurrency: 1
         containers:
         - image: registry-vpc.cn-hangzhou.aliyuncs.com/hz-suoxing-test/test:helloworld-go
           name: user-container
           ports:
           - containerPort: 6666
             name: http1
             protocol: TCP
           resources:
             limits:
               aliyun.com/gpu-mem: "3"    # Specify the GPU memory size.

Next steps

• Best practices for deploying AI inference services in Knative — deploy AI models, configure autoscaling, and allocate GPU resources.

• GPU FAQs