AI node storage usage

Updated at:

This topic describes the storage options supported by PolarDB-X AI nodes, helping you choose the appropriate storage solution for different requirements such as model files, shared data, persistent data, and temporary cache. Pods on AI nodes can mount storage through Kubernetes PersistentVolume (PV), PersistentVolumeClaim (PVC), and emptyDir. Storage resources must match the network environment of the AI node, including the Kubernetes cluster, VPC, and region.

Supported storage options

Storage option

Use case

Access mode

Data lifecycle

ESSD cloud disk

Model caching, intermediate inference results, KV Cache, and data that requires persistent read/write

Typically suitable for a single Pod or a small number of Pods

Cloud disks exist independently of Pods and are not released by default after the Pod is deleted.

CPFS

Large model weights, high-throughput shared files, and shared read/write across multiple Pods

Multiple Pods can share access

Data is stored in the CPFS file system and is not deleted when Pods are removed.

OSS Bucket

Model weights, configuration files, shared read-only data, and archived data

Multiple Pods can share access

Data is stored in OSS and is not deleted when Pods are removed.

NAS

General shared files, configuration files, and lightweight shared read/write data

Multiple Pods can share access

Data is stored in the NAS file system and is not deleted when Pods are removed.

Local ephemeral storage

Temporary compute files and scratch data

In-Pod access

Follows the Pod lifecycle and is released when the Pod is deleted.

Memory-backed temporary directory

/dev/shm, high-frequency temporary read/write, and small file caching

In-Pod access

Uses available Pod memory and is released when the Pod is deleted.

Selection recommendations

Requirement

Recommended solution

High-performance persistent block storage required

ESSD cloud disk

Multiple inference Pods sharing large model weights with high throughput requirements

CPFS

Multiple Pods sharing model files, configuration files, or archived files

OSS Bucket

General shared file system capabilities required

NAS

Only temporary compute files need to be stored

Local ephemeral storage

Need to mount /dev/shm or use a memory-backed temporary directory

Memory-backed temporary directory

Limits

  • Storage resources must be network-connected to the cluster where the AI node resides. We recommend that you select storage resources in the same region and VPC or accessible over the internal network.

  • ESSD cloud disks, CPFS, OSS, and NAS all require creating a PVC first and then referencing the PVC in the Pod.

  • The performance and consistency semantics of shared storage such as OSS, NAS, and CPFS are determined by the corresponding cloud product. Select based on your business read/write model.

  • Local ephemeral storage and memory-backed temporary directories do not provide persistence. Do not store model weights, business results, or other data that requires long-term retention.

Use ephemeral storage

If you only need to store temporary compute files, you can use Kubernetes emptyDir.

apiVersion: v1
kind: Pod
metadata:
  name: ai-node-scratch-example
  namespace: default
spec:
  volumes:
    - name: scratch-volume
      emptyDir: {}
  containers:
    - name: app
      image: <image>
      volumeMounts:
        - mountPath: /scratch
          name: scratch-volume

If your application requires /dev/shm, you can mount a memory-backed temporary directory by setting emptyDir.medium: Memory.

apiVersion: v1
kind: Pod
metadata:
  name: ai-node-shm-example
  namespace: default
spec:
  volumes:
    - name: shm-volume
      emptyDir:
        medium: Memory
        sizeLimit: 32Gi
  containers:
    - name: app
      image: <image>
      volumeMounts:
        - mountPath: /dev/shm
          name: shm-volume

Set sizeLimit based on application concurrency and memory consumption during model execution to avoid excessive memory usage that affects Pod stability.