Mount cloud disks
This topic describes how to mount an ESSD cloud disk to a PolarDB-X AI node Pod and verify the mounting, read/write, and data persistence capabilities of the cloud disk. Cloud disks can be used for persistent storage scenarios such as model files, model caching, intermediate inference results, and KV Cache.
Preparations
Make sure that the following conditions are met:
-
A PolarDB-X AI node is created and in the normal state.
-
The cloud disk mounting capability is enabled on the AI node.
-
A PolarDB-X AI node cloud disk storage plan is purchased, and the remaining capacity is not less than the capacity requested by the cloud disk.
-
The credentials of a Kubernetes cluster that can access the AI node are configured, and you have the permissions to create Pods and PVCs.
-
The Pod, AI node, and cloud disk in the ACK cluster are in the same zone.
The examples in this topic use the following parameters. Replace them based on your actual environment:
|
Parameter |
Example |
Description |
|
AI node ID |
|
Replace with the actual AI node ID. |
|
Business image |
|
Replace with a business image that can start properly. |
|
Namespace |
|
Replace as needed. |
|
Cloud disk capacity |
|
The capacity cannot exceed the remaining capacity of the storage plan. |
Usage notes
-
ESSD cloud disks are non-shared storage. We recommend that you use
ReadWriteOnceand mount each cloud disk to only one Pod at a time. -
Deleting a Pod does not delete the PVC or the cloud disk, and the cloud disk capacity is not immediately returned to the storage plan.
-
After deleting the PVC, verify that the storage plan capacity has been returned to avoid continued quota occupation or charges.
-
When a cloud disk is no longer needed, make sure that the data on the cloud disk is backed up before deleting the PVC. Deleting the PVC may make the cloud disk data inaccessible.
-
Select the cloud disk capacity based on the model size and actual read/write load.
Procedure
Step 1: Verify the AI node state
Run the following command to view the nodes:
kubectl get nodes
Make sure that the target AI node is in the Ready state. If GPUs are required for your business, also verify that the node has sufficient allocatable GPU resources.
Step 2: Create a PVC
AI node cloud disks use the following StorageClass:
aliclouddb-vnode-disk-essd
-
Create a file named
ai-node-disk-pvc.yaml:apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ai-node-disk namespace: default spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi storageClassName: aliclouddb-vnode-disk-essd volumeMode: FilesystemTo request a different capacity, modify
resources.requests.storage. The requested capacity cannot exceed the remaining capacity of the cloud disk storage plan. -
Apply the PVC:
kubectl apply -f ai-node-disk-pvc.yaml kubectl get pvc ai-node-disk -n defaultThe expected PVC status is
Bound. You can also run the following command to view the details:kubectl describe pvc ai-node-disk -n defaultNoteAI node cloud disks are provided by the platform. A Kubernetes StorageClass object with the same name may not exist in the ACK cluster. Use whether the PVC enters the
Boundstate as the criterion for successful cloud disk binding.
Step 3: Create a Pod and mount the PVC
-
Create a file named
ai-node-disk-pod.yamland replace the following content:-
Replace
<AI_NODE_ID>with the target AI node ID. -
Replace
<IMAGE>with your business image. -
If you use a different namespace, modify the namespace in the Pod, PVC, and commands accordingly.
apiVersion: v1 kind: Pod metadata: name: ai-node-disk-example namespace: default spec: restartPolicy: Never nodeSelector: rm.alibaba-inc.com/vnodeId: <AI_NODE_ID> tolerations: - key: virtual-kubelet.io/provider operator: Equal value: aliclouddb effect: NoSchedule containers: - name: app image: <IMAGE> command: - /bin/sh - -c - sleep 3600 resources: requests: cpu: "1" memory: 2Gi limits: cpu: "1" memory: 2Gi volumeMounts: - name: data-disk mountPath: /data volumes: - name: data-disk persistentVolumeClaim: claimName: ai-node-diskIf your business requires GPUs, add the following line to both
resources.requestsandresources.limits:nvidia.com/gpu: "1" -
-
Apply the Pod and check the scheduling result:
kubectl apply -f ai-node-disk-pod.yaml kubectl get pod ai-node-disk-example -n default -o wideMake sure that the Pod is in the
Runningstate and that theNODEcolumn shows the target AI node.
Verify the cloud disk
Verify that the cloud disk is mounted
Run the following commands to view the mounted device and file system:
kubectl exec ai-node-disk-example -n default -- findmnt -T /data
kubectl exec ai-node-disk-example -n default -- df -hT /data
If the output includes an independent block device, a file system type, and the /data mount point, the cloud disk is mounted.
Verify that the cloud disk is readable and writable
Run the following command to write and read a test file:
kubectl exec ai-node-disk-example -n default -- sh -c \
'echo ai-node-disk-test > /data/persist-marker.txt && sync && cat /data/persist-marker.txt'
If ai-node-disk-test is returned, the cloud disk supports normal read/write operations.
Verify that data persists after the Pod is rebuilt
-
Delete the test Pod. Deleting the Pod does not delete the PVC:
kubectl delete pod ai-node-disk-example -n default -
Recreate the Pod:
kubectl apply -f ai-node-disk-pod.yaml kubectl wait --for=condition=Ready pod/ai-node-disk-example -n default --timeout=300s -
Read the test file:
kubectl exec ai-node-disk-example -n default -- cat /data/persist-marker.txtIf
ai-node-disk-testis still returned, the data persists after the Pod is rebuilt.
FAQ
The PVC remains in the Pending state
Check the following items in order:
-
Whether a cloud disk storage plan is purchased.
-
Whether the remaining capacity of the storage plan is sufficient.
-
Whether the cloud disk mounting capability is enabled on the AI node.
-
Whether
storageClassNameis set toaliclouddb-vnode-disk-essd. -
Whether the PVC and the target AI node are in the same zone.
-
Whether PVC events indicate insufficient capacity, invalid parameters, or permission errors.
View the details:
kubectl describe pvc ai-node-disk -n default
The Pod is not scheduled to the target AI node
Check the following items:
-
Whether
rm.alibaba-inc.com/vnodeIdis set to the correct AI node ID. -
Whether a toleration for the
aliclouddbprovider is configured. -
Whether the target AI node is in the
Readystate. -
Whether the node has sufficient CPU, memory, and GPU resources to meet the Pod requests.
-
Whether the Pod and the cloud disk are in the same zone.
View the Pod and node details:
kubectl describe pod ai-node-disk-example -n default
kubectl get node <AI_NODE_ID>