Use local disk volumes for GPU-HPN capacity reservation

Step 1: Check for available NVMe disks

Before creating a PVC, ensure your cluster has available local NVMe disks.

1. Run the following command to check for available local NVMe disks on each node:

   kubectl get csistoragecapacities -n kube-system \
     -l csi.storage.k8s.io/drivername=local-passthrough.csi.alibabacloud.com \
     -o jsonpath='{range .items[?(@.capacity!="0")]}{.metadata.name}{"\t"}{.nodeTopology.matchLabels.csi\.alibabacloud\.com/node-name}{"\t"}{.maximumVolumeSize}{"\t"}{.capacity}{"\n"}{end}' \
     | awk 'BEGIN{printf "%-14s %-50s %-16s %s\n","NAME","NODE","DISK-SIZE","AVAILABLE"} {split($3,a,"Ki");split($4,b,"Ki");printf "%-14s %-50s %-16s %d\n",$1,$2,$3,b[1]/a[1]}'

2. Check the expected output:

   NAME           NODE                                               DISK-SIZE        AVAILABLE
   csisc-12345    cn-wulanchabu-c.cr-ovt2bz1uoiyoxxxxxxxxx           3750738264Ki     2
   csisc-abcde    cn-wulanchabu-b.cr-xyz123456789xxxxxxxxx           3750738264Ki     1

   • DISK-SIZE: The capacity of a single disk (maximumVolumeSize).

   • AVAILABLE: The number of remaining allocatable disks on the node (capacity / maximumVolumeSize).

   Note

   If AVAILABLE is 0 for all nodes, all disks are in use. You must wait for other PVCs to be released or scale out the nodes.

Step 2: Create a PVC

1. Create a file named pvc.yaml that uses the local-passthrough StorageClass. Installing the acs-local-passthrough-controller component creates the local-passthrough StorageClass by default. Its reclaim policy is reclaimPolicy: Delete and its binding mode is WaitForFirstConsumer.

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: local-disk-pvc
     namespace: default
   spec:
     accessModes:
       - ReadWriteOncePod    # Exclusive access by a single Pod.
     volumeMode: Block       # Only the block device mode is supported.
     storageClassName: local-passthrough
     resources:
       requests:
         storage: 10Gi       # Must be less than or equal to the DISK-SIZE from Step 1. The entire physical disk is allocated.

2. Apply the configuration:

   kubectl apply -f pvc.yaml

   Because the StorageClass uses the WaitForFirstConsumer binding mode, the PVC remains in the Pending state until a Pod references it.

   Note

   If the requested storage value exceeds the DISK-SIZE from Step 1, the PVC creation fails.

Step 3: Create a GPU Pod and mount the disk

1. Create pod.yaml and use volumeDevices to attach a block device:

   apiVersion: v1
   kind: Pod
   metadata:
     name: gpu-workload
     namespace: default
     labels:
       alibabacloud.com/compute-class: "gpu-hpn"   # Required. Specifies the GPU-HPN compute type.
   spec:
     containers:
       - name: main
         # Example: alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/alinux3:latest
         image: <your-gpu-image>
         volumeDevices:                            # Use volumeDevices for block devices.
           - name: local-disk
             devicePath: /dev/xvda
     volumes:
       - name: local-disk
         persistentVolumeClaim:
           claimName: local-disk-pvc

2. Apply the configuration:

   kubectl apply -f pod.yaml

Step 4: Verify the mount and data persistence

1. Confirm that the PVC is bound:

   kubectl get pvc local-disk-pvc

   Expected output:

   NAME             STATUS   VOLUME                                                  CAPACITY       ACCESS MODES   STORAGECLASS        AGE
   local-disk-pvc   Bound    local-passthrough-d0cfcee6-734d-4d59-ae76-xxxxxxxxxxxx  3750738264Ki   RWOP           local-passthrough   1m

   A STATUS of Bound confirms that the PVC is bound to a local NVMe disk and ready for use.

2. Confirm that the block device is visible inside the Pod:

   kubectl exec gpu-workload -- lsblk

3. Write data to the block device and then read it back:

   # Write data to the block device.
   kubectl exec gpu-workload -- sh -c 'echo "test" > /dev/xvda'
   
   # Read the data to verify.
   kubectl exec gpu-workload -- head -c 5 /dev/xvda

   Expected output: test

4. Verify that data persists after Pod recreation (delete only the Pod, not the PVC):

   # Delete the Pod but keep the PVC.
   kubectl delete pod gpu-workload
   
   # Recreate the Pod by using the same configuration.
   kubectl apply -f pod.yaml
   
   # Wait for the Pod to be ready, then read the data.
   kubectl wait --for=condition=ready pod/gpu-workload --timeout=60s
   kubectl exec gpu-workload -- head -c 5 /dev/xvda

   The expected output is still: test

Why are my Pod and PVC stuck in the Pending state?

In WaitForFirstConsumer mode, PVC binding depends on successful Pod scheduling. First investigate why the Pod is failing to schedule.

kubectl describe pod gpu-workload

Common causes:

Why do I get an "insufficient space" error when writing large files?

The actual writable capacity is the full physical capacity of the disk (DISK-SIZE), regardless of the storage request value in the PVC.

How can I make my data persistent?

Local disks do not provide a persistence guarantee. For data that requires persistence, use NAS, OSS, or CPFS volumes.