Mount cloud disks

Updated at:

This topic describes how to mount an ESSD cloud disk to a PolarDB-X AI node Pod and verify the mounting, read/write, and data persistence capabilities of the cloud disk. Cloud disks can be used for persistent storage scenarios such as model files, model caching, intermediate inference results, and KV Cache.

Preparations

Make sure that the following conditions are met:

  • A PolarDB-X AI node is created and in the normal state.

  • The cloud disk mounting capability is enabled on the AI node.

  • A PolarDB-X AI node cloud disk storage plan is purchased, and the remaining capacity is not less than the capacity requested by the cloud disk.

  • The credentials of a Kubernetes cluster that can access the AI node are configured, and you have the permissions to create Pods and PVCs.

  • The Pod, AI node, and cloud disk in the ACK cluster are in the same zone.

The examples in this topic use the following parameters. Replace them based on your actual environment:

Parameter

Example

Description

AI node ID

<AI_NODE_ID>

Replace with the actual AI node ID.

Business image

<IMAGE>

Replace with a business image that can start properly.

Namespace

default

Replace as needed.

Cloud disk capacity

100Gi

The capacity cannot exceed the remaining capacity of the storage plan.

Usage notes

  • ESSD cloud disks are non-shared storage. We recommend that you use ReadWriteOnce and mount each cloud disk to only one Pod at a time.

  • Deleting a Pod does not delete the PVC or the cloud disk, and the cloud disk capacity is not immediately returned to the storage plan.

  • After deleting the PVC, verify that the storage plan capacity has been returned to avoid continued quota occupation or charges.

  • When a cloud disk is no longer needed, make sure that the data on the cloud disk is backed up before deleting the PVC. Deleting the PVC may make the cloud disk data inaccessible.

  • Select the cloud disk capacity based on the model size and actual read/write load.

Procedure

Step 1: Verify the AI node state

Run the following command to view the nodes:

kubectl get nodes

Make sure that the target AI node is in the Ready state. If GPUs are required for your business, also verify that the node has sufficient allocatable GPU resources.

Step 2: Create a PVC

AI node cloud disks use the following StorageClass:

aliclouddb-vnode-disk-essd
  1. Create a file named ai-node-disk-pvc.yaml:

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: ai-node-disk
      namespace: default
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi
      storageClassName: aliclouddb-vnode-disk-essd
      volumeMode: Filesystem

    To request a different capacity, modify resources.requests.storage. The requested capacity cannot exceed the remaining capacity of the cloud disk storage plan.

  2. Apply the PVC:

    kubectl apply -f ai-node-disk-pvc.yaml
    kubectl get pvc ai-node-disk -n default

    The expected PVC status is Bound. You can also run the following command to view the details:

    kubectl describe pvc ai-node-disk -n default
    Note

    AI node cloud disks are provided by the platform. A Kubernetes StorageClass object with the same name may not exist in the ACK cluster. Use whether the PVC enters the Bound state as the criterion for successful cloud disk binding.

Step 3: Create a Pod and mount the PVC

  1. Create a file named ai-node-disk-pod.yaml and replace the following content:

    • Replace <AI_NODE_ID> with the target AI node ID.

    • Replace <IMAGE> with your business image.

    • If you use a different namespace, modify the namespace in the Pod, PVC, and commands accordingly.

    apiVersion: v1
    kind: Pod
    metadata:
      name: ai-node-disk-example
      namespace: default
    spec:
      restartPolicy: Never
      nodeSelector:
        rm.alibaba-inc.com/vnodeId: <AI_NODE_ID>
      tolerations:
        - key: virtual-kubelet.io/provider
          operator: Equal
          value: aliclouddb
          effect: NoSchedule
      containers:
        - name: app
          image: <IMAGE>
          command:
            - /bin/sh
            - -c
            - sleep 3600
          resources:
            requests:
              cpu: "1"
              memory: 2Gi
            limits:
              cpu: "1"
              memory: 2Gi
          volumeMounts:
            - name: data-disk
              mountPath: /data
      volumes:
        - name: data-disk
          persistentVolumeClaim:
            claimName: ai-node-disk

    If your business requires GPUs, add the following line to both resources.requests and resources.limits:

    nvidia.com/gpu: "1"
  2. Apply the Pod and check the scheduling result:

    kubectl apply -f ai-node-disk-pod.yaml
    kubectl get pod ai-node-disk-example -n default -o wide

    Make sure that the Pod is in the Running state and that the NODE column shows the target AI node.

Verify the cloud disk

Verify that the cloud disk is mounted

Run the following commands to view the mounted device and file system:

kubectl exec ai-node-disk-example -n default -- findmnt -T /data
kubectl exec ai-node-disk-example -n default -- df -hT /data

If the output includes an independent block device, a file system type, and the /data mount point, the cloud disk is mounted.

Verify that the cloud disk is readable and writable

Run the following command to write and read a test file:

kubectl exec ai-node-disk-example -n default -- sh -c \
  'echo ai-node-disk-test > /data/persist-marker.txt && sync && cat /data/persist-marker.txt'

If ai-node-disk-test is returned, the cloud disk supports normal read/write operations.

Verify that data persists after the Pod is rebuilt

  1. Delete the test Pod. Deleting the Pod does not delete the PVC:

    kubectl delete pod ai-node-disk-example -n default
  2. Recreate the Pod:

    kubectl apply -f ai-node-disk-pod.yaml
    kubectl wait --for=condition=Ready pod/ai-node-disk-example -n default --timeout=300s
  3. Read the test file:

    kubectl exec ai-node-disk-example -n default -- cat /data/persist-marker.txt

    If ai-node-disk-test is still returned, the data persists after the Pod is rebuilt.

FAQ

The PVC remains in the Pending state

Check the following items in order:

  • Whether a cloud disk storage plan is purchased.

  • Whether the remaining capacity of the storage plan is sufficient.

  • Whether the cloud disk mounting capability is enabled on the AI node.

  • Whether storageClassName is set to aliclouddb-vnode-disk-essd.

  • Whether the PVC and the target AI node are in the same zone.

  • Whether PVC events indicate insufficient capacity, invalid parameters, or permission errors.

View the details:

kubectl describe pvc ai-node-disk -n default

The Pod is not scheduled to the target AI node

Check the following items:

  • Whether rm.alibaba-inc.com/vnodeId is set to the correct AI node ID.

  • Whether a toleration for the aliclouddb provider is configured.

  • Whether the target AI node is in the Ready state.

  • Whether the node has sufficient CPU, memory, and GPU resources to meet the Pod requests.

  • Whether the Pod and the cloud disk are in the same zone.

View the Pod and node details:

kubectl describe pod ai-node-disk-example -n default
kubectl get node <AI_NODE_ID>