Configure automatic disk expansion

更新时间:
复制 MD 格式

Set thresholds in a StorageAutoScalerPolicy so storage-auto-expander in storage-operator auto-expands disk volumes.

How it works

When storage-auto-expander detects that PVC disk usage exceeds the threshold, it expands the volume in this sequence:

  1. Monitors disk usage on PVCs matching the policy's label selector.

  2. When usage exceeds the threshold, selects the first matching action and submits a resize request to the CSI (Container Storage Interface) driver.

  3. The CSI driver resizes the cloud disk and signals Kubernetes to resize the file system.

  4. PVC capacity updates after the file system resize completes.

Timing constraints: The trigger check interval is at most 2 minutes, and expansion takes about 1 minute. Do not fill the disk within 3 minutes of a previous expansion.

Limitations

Limitation Detail
Kubernetes version Requires Kubernetes 1.16 or later. Upgrade the cluster if needed.
Disk type Basic disks are not supported. See ResizeDisk for eligible disk types.
StorageClass The PVC's StorageClass must have allowVolumeExpansion: true. ACK-provided StorageClasses enable this by default. For custom StorageClasses, set the parameter at creation time — existing StorageClasses cannot be modified.
Pod state The application pod using the disk must be in Running state.

Enable the storage-auto-expander module

The storage-auto-expander module in storage-operator handles automatic expansion. To check whether storage-operator is installed and find its version, go to Operations > Add-ons > Volumes.

  • v1.33.1 and later: The module is enabled by default. No action needed.

  • Earlier than v1.33.1: Enable the module:

    kubectl patch configmap/storage-operator \
      -n kube-system \
      --type merge \
      -p '{"data":{"storage-auto-expander":"{\"imageRep\":\"acs/storage-auto-expander\",\"imageTag\":\"\",\"install\":\"true\",\"template\":\"/acs/templates/storage-auto-expander/install.yaml\",\"type\":\"deployment\"}"}}'

    To upgrade storage-operator, see Upgrade add-ons.

Configure an automatic expansion policy

Step 1: Verify that volume expansion is enabled on your StorageClass

kubectl get sc

The ALLOWVOLUMEEXPANSION column must be true. Example output:

NAME                                    PROVISIONER                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
alicloud-disk-efficiency                diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   26h
alicloud-disk-essd                      diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   26h
alicloud-disk-ssd                       diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   26h
alicloud-disk-topology-alltype          diskplugin.csi.alibabacloud.com   Delete          WaitForFirstConsumer   true                   26h
Use alicloud-disk-topology-alltype when possible. It auto-selects the disk type based on instance type and zone availability, reducing provisioning failures. See Use dynamically provisioned volumes.

Step 2: Create a StorageAutoScalerPolicy

Create a file named StorageAutoScalerPolicy.yaml:

apiVersion: storage.alibabacloud.com/v1alpha1
kind: StorageAutoScalerPolicy
metadata:
  name: hybrid-expand-policy
spec:
  pvcSelector:
    matchLabels:
      app: nginx
  namespaces:
    - default
    - nginx
  conditions:
    - name: condition1
      key: volume-capacity-used-percentage
      operator: Gt
      values:
        - "80"
  actions:
    - name: action1
      type: volume-expand
      params:
        scale: 50Gi
        limits: 100Gi
    - name: action2
      type: volume-expand
      params:
        scale: 50%
        limits: 300Gi

Key fields:

Field Description
pvcSelector Selects PVCs by label. Example: app: nginx.
namespaces Target PVC namespaces. Multiple values use OR logic. Default: default.
conditions Trigger conditions. Multiple conditions use AND logic. key: metric (volume-capacity-used-percentage = disk usage percentage). operator: Gt (greater than), Lt (less than), Eq (equal), or Ne (not equal) — case-insensitive. values: threshold as a string.
actions Actions to run when conditions are met. The first matching action runs; the rest are skipped. type: must be volume-expand. params.scale: expansion amount — absolute (e.g., 50Gi) or percentage (e.g., 50%). params.limits: maximum PVC size for this action.

The policy triggers when PVC usage exceeds 80%:

  • action1 adds 50 GiB (max 100 GiB). Applies while PVC is under 100 GiB.

  • action2 adds 50% of current size (max 300 GiB). Applies once PVC reaches 100 GiB.

Step 3: Apply the policy

kubectl create -f StorageAutoScalerPolicy.yaml

Verify automatic expansion

The following steps trigger five expansion events with a test StatefulSet, using the policy defined above.

Step 1: Create a test StatefulSet

Create a file named StatefulSet.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
        ports:
        - containerPort: 80
        volumeMounts:
        - name: pvc-disk
          mountPath: /data
      volumes:
        - name: pvc-disk
          persistentVolumeClaim:
            claimName: disk-pvc
  volumeClaimTemplates:
    - metadata:
        name: pvc-disk
        labels:
          app: nginx
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "alicloud-disk-topology-alltype"
        resources:
          requests:
            storage: 25Gi

Apply it and confirm the pod is running:

kubectl create -f StatefulSet.yaml
kubectl get pod -l app=nginx

Expected output:

NAME      READY   STATUS    RESTARTS   AGE
nginx-0   1/1     Running   0          99s

Check the initial disk capacity:

kubectl exec -it nginx-0 -- df -h /data

Expected output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb         25G   24K   25G   1% /data

Step 2: Trigger the first expansion (25 GiB to 75 GiB)

Write 22 GB of data to push usage above 80%:

kubectl exec -it nginx-0 -- fallocate -l 22G /data/test1

Check expansion events:

kubectl get events

The disk is 25 GiB and usage exceeds 80%, so action1 adds 50 GiB:

2m1s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 25Gi to 75Gi, usedCapacityPercentage:90%, freeSize:2498MB.
2m1s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
2m1s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
118s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node
116s   Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0    Pvc pvc-disk-nginx-0 is expanding status from 25Gi to 75Gi, this action action2 will skip.

Verify the PVC capacity:

kubectl get pvc

Expected output:

NAME               STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS                     VOLUMEATTRIBUTESCLASS   AGE
pvc-disk-nginx-0   Bound    d-uf66kkzltnq6xgi9****   75Gi       RWO            alicloud-disk-topology-alltype   <unset>                 26m

Step 3: Trigger the second expansion (75 GiB to 100 GiB)

Write 40 GB to push usage above 80% again:

kubectl exec -it nginx-0 -- fallocate -l 40G /data/test2

action1 fires again, but its 100 GiB limit caps expansion at 100 GiB:

kubectl get events
7m4s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0      Start to expand of pvc pvc-disk-nginx-0 from 75Gi to 100Gi, usedCapacityPercentage:84%, freeSize:11927MB.
7m4s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0      waiting for an external controller to expand this PVC
7m4s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0      External resizer is resizing volume d-uf66kkzltnq6xgi9****
7m1s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0      Require file system resize of volume on node
5m59s  Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0      Pvc pvc-disk-nginx-0 is expanding status from 75Gi to 100Gi, this action action2 will skip.

Step 4: Trigger the third expansion (100 GiB to 150 GiB)

At 100 GiB, action1's limit is reached, so action2 takes over and adds 50% of current size:

kubectl exec -it nginx-0 -- fallocate -l 20G /data/test3
kubectl get events
2m40s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 100Gi to 150Gi, usedCapacityPercentage:83%, freeSize:16637MB.
2m40s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
2m40s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
2m37s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node
109s    Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0    Pvc pvc-disk-nginx-0 is expanding status from 100Gi to 150Gi, this action action2 will skip.

Step 5: Trigger the fourth expansion (150 GiB to 225 GiB)

action2 fires again, adding 50% of 150 GiB = 75 GiB:

kubectl exec -it nginx-0 -- fallocate -l 50G /data/test4
kubectl get events
2m42s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 150Gi to 225Gi, usedCapacityPercentage:87%, freeSize:19621MB.
2m42s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
2m42s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
2m38s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node
114s    Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0    Pvc pvc-disk-nginx-0 is expanding status from 150Gi to 225Gi, this action action2 will skip.

Step 6: Trigger the fifth expansion (225 GiB to 300 GiB)

action2 fires one final time, capped at its 300 GiB limit:

kubectl exec -it nginx-0 -- fallocate -l 50G /data/test5
kubectl get events
17m     Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 225Gi to 300Gi, usedCapacityPercentage:82%, freeSize:40351MB.
17m     Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
17m     Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
17m     Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node

Both actions have reached their limits. No further expansion occurs.

Next steps

For cloud disk volume issues, see Cloud disk volume FAQ.