DocsICP FilingConsole
官方文档
首页

Configure automatic expansion for local volumes

更新时间：
复制 MD 格式
产品详情
我的收藏

Local volumes let you use ECS local disks or cloud disks to store temporary data that spikes during peak hours and shrinks off-peak. ACK schedules local storage resources based on per-node capacity, and the Storage Auto Expander automatically grows a volume when usage crosses a configured threshold.

How it works

The local storage stack has three components that work together end-to-end:

Component Role
Node Resource Manager Reads a ConfigMap to initialize VolumeGroups and QuotaPaths on each node
CSI plug-in Manages the full lifecycle of local volumes: creation, mounting, unmounting, deletion, and expansion
Storage Auto Expander Monitors volume usage and triggers automatic expansion when capacity falls below the configured threshold
image

Failure behavior: Local volumes are tied to the node where they reside. If the node becomes unhealthy, the volume becomes inaccessible and any pods using it cannot run. Applications using local volumes must tolerate this reduced availability.

Limitations

These constraints affect architecture decisions — review them before you begin:

Constraint Detail
Supported volume types hostPath, local, LV (logical volume), and memory
No high availability Local volumes are node-bound. If the node fails, the volume becomes inaccessible. Use local volumes only for temporary storage or when your application provides its own HA.
LVs cannot be migrated A logical volume (LV) is bound to the physical disks on a specific node. If the node fails and the pod is rescheduled, the LV is no longer accessible on the new node. LVs are not suitable for HA scenarios.

Prerequisites

Before you begin, make sure you have:

Step 1: Initialize a local volume

Node Resource Manager reads the node-resource-topo ConfigMap in the kube-system namespace to initialize local storage on each node. The ConfigMap defines two types of resources: VolumeGroups (for LVM-backed logical volumes) and QuotaPaths (for quota-enforced directory mounts).

1.1 Define local volume resources in the ConfigMap

The value field in the ConfigMap must match the target node name — the same name you will reference in PVC annotations in Step 2.

Replace cn-zhangjiakou.192.168.XX.XX with the actual node name.

Apply ConfigMap

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-resource-topo
  namespace: kube-system
data:
  volumegroup: |-
    volumegroup:
    - name: volumegroup1
      key: kubernetes.io/hostname
      operator: In
      value: cn-zhangjiakou.192.168.XX.XX
      topology:
        type: device
        devices:
        - /dev/vdb
        - /dev/vdc
  quotapath: |-
    quotapath:
    - name: /mnt/path1
      key: kubernetes.io/hostname
      operator: In
      value: cn-zhangjiakou.192.168.XX.XX
      topology:
        type: device
        options: prjquota
        fstype: ext4
        devices:
        - /dev/vdd
EOF

This ConfigMap defines two resources on the cn-zhangjiakou.192.168.XX.XX node:

  • VolumeGroup (volumegroup1): Backed by block devices /dev/vdb and /dev/vdc. The CSI plug-in carves logical volumes from this group when a PVC is created.

  • QuotaPath (/mnt/path1): /dev/vdd is formatted with ext4 and mounted to /mnt/path1 with project quota (prjquota) enabled. Only one devices entry is allowed per QuotaPath.

1.2 Deploy the Node Resource Manager

Node Resource Manager runs as a DaemonSet on every node, watches the ConfigMap, and applies the declared storage topology.

Create the nrm.yaml file

cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: node-resource-manager
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: node-resource-manager
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "watch", "list", "delete", "update", "create"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: node-resource-manager-binding
subjects:
  - kind: ServiceAccount
    name: node-resource-manager
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: node-resource-manager
  apiGroup: rbac.authorization.k8s.io
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: node-resource-manager
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: node-resource-manager
  template:
    metadata:
      labels:
        app: node-resource-manager
    spec:
      tolerations:
        - operator: "Exists"
      priorityClassName: system-node-critical
      serviceAccountName: node-resource-manager
      hostNetwork: true
      hostPID: true
      containers:
        - name: node-resource-manager
          securityContext:
            privileged: true
            capabilities:
              add: ["SYS_ADMIN"]
            allowPrivilegeEscalation: true
          image: registry.cn-hangzhou.aliyuncs.com/acs/node-resource-manager:v1.18.8.0-5b1bdc2-aliyun
          imagePullPolicy: "Always"
          args:
            - "--nodeid=$(KUBE_NODE_NAME)"
          env:
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
          volumeMounts:
            - mountPath: /dev
              mountPropagation: "HostToContainer"
              name: host-dev
            - mountPath: /var/log/
              name: host-log
            - name: etc
              mountPath: /host/etc
            - name: config
              mountPath: /etc/unified-config
      volumes:
        - name: host-dev
          hostPath:
            path: /dev
        - name: host-log
          hostPath:
            path: /var/log/
        - name: etc
          hostPath:
            path: /etc
        - name: config
          configMap:
            name: node-resource-topo
EOF

1.3 Install the CSI plug-in

The CSI plug-in (csi-local-plugin) manages the complete data volume lifecycle: creation, mounting, unmounting, deletion, and expansion.

cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: localplugin.csi.alibabacloud.com
spec:
  attachRequired: false
  podInfoOnMount: true
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: csi-local-plugin
  name: csi-local-plugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-local-plugin
  template:
    metadata:
      labels:
        app: csi-local-plugin
    spec:
      containers:
      - args:
        - --v=5
        - --csi-address=/csi/csi.sock
        - --kubelet-registration-path=/var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com/csi.sock
        env:
        - name: KUBE_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        image: registry.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v1.2.0
        imagePullPolicy: Always
        name: driver-registrar
        volumeMounts:
        - mountPath: /csi
          name: plugin-dir
        - mountPath: /registration
          name: registration-dir
      - args:
        - --endpoint=$(CSI_ENDPOINT)
        - --v=5
        - --nodeid=$(KUBE_NODE_NAME)
        - --driver=localplugin.csi.alibabacloud.com
        env:
        - name: KUBE_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: SERVICE_PORT
          value: "11290"
        - name: CSI_ENDPOINT
          value: unix://var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com/csi.sock
        image: registry.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.14.8-40ee9518-local
        imagePullPolicy: Always
        name: csi-localplugin
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            add:
            - SYS_ADMIN
          privileged: true
        volumeMounts:
        - mountPath: /var/lib/kubelet
          mountPropagation: Bidirectional
          name: pods-mount-dir
        - mountPath: /dev
          mountPropagation: HostToContainer
          name: host-dev
        - mountPath: /var/log/
          name: host-log
        - mountPath: /mnt
          mountPropagation: Bidirectional
          name: quota-path-dir
      hostNetwork: true
      hostPID: true
      serviceAccount: csi-admin
      tolerations:
      - operator: Exists
      volumes:
      - hostPath:
          path: /var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com
          type: DirectoryOrCreate
        name: plugin-dir
      - hostPath:
          path: /var/lib/kubelet/plugins_registry
          type: DirectoryOrCreate
        name: registration-dir
      - hostPath:
          path: /var/lib/kubelet
          type: Directory
        name: pods-mount-dir
      - hostPath:
          path: /dev
          type: ""
        name: host-dev
      - hostPath:
          path: /var/log/
          type: ""
        name: host-log
      - hostPath:
          path: /mnt
          type: Directory
        name: quota-path-dir
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%
    type: RollingUpdate
EOF

Verify step 1

Confirm both DaemonSets are running before proceeding:

kubectl get daemonset -n kube-system | grep -E 'node-resource-manager|csi-local-plugin'

Expected output:

node-resource-manager   1         1         1       1            1           <none>          ...
csi-local-plugin        1         1         1       1            1           <none>          ...

Both DaemonSets should show all pods ready (desired = ready).

Step 2: Create an application using a local volume

Two volume types are available. Choose based on your workload:

Volume type Backed by Use when
LVM (LV) VolumeGroup (LVM logical volumes) Workloads needing block storage with flexible sizing
QuotaPath Formatted disk with project quota Workloads needing directory-based quota enforcement

Create an application using an LVM local volume

2.1 Create a StorageClass

The vgName parameter must match the VolumeGroup name defined in the node-resource-topo ConfigMap — volumegroup1 in this example.

volumeBindingMode: WaitForFirstConsumer delays PVC binding until a pod is scheduled. Without this, the PVC could bind to a volume on the wrong node before the scheduler determines where the pod runs, making the pod unschedulable.

cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
    name: csi-local
provisioner: localplugin.csi.alibabacloud.com
parameters:
    volumeType: LVM
    vgName: volumegroup1
    fsType: ext4
    lvmType: "striping"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF

For more information, see Use LVs.

2.2 Create a PVC

The volume.kubernetes.io/selected-node annotation pins the PVC to the node where the VolumeGroup is initialized. Set it to the same node name used in the ConfigMap value field.

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lvm-pvc
  annotations:
    volume.kubernetes.io/selected-node: cn-zhangjiakou.192.168.XX.XX
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: csi-local
EOF

2.3 Deploy the application

Create a sample application

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-lvm
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
        volumeMounts:
          - name: lvm-pvc
            mountPath: "/data"
      volumes:
        - name: lvm-pvc
          persistentVolumeClaim:
            claimName: lvm-pvc
EOF

Create an application using a QuotaPath local volume

2.4 Create a StorageClass

The rootPath parameter must match the QuotaPath mount path defined in the node-resource-topo ConfigMap — /mnt/path1 in this example.

cat << EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-local-quota
parameters:
  volumeType: QuotaPath
  rootPath: /mnt/path1
provisioner: localplugin.csi.alibabacloud.com
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
EOF

For more information, see Use QuotaPath volumes.

2.5 Create a PVC

The app: web-quota label is required — the auto-expansion policy in Step 3 uses this label to select PVCs for monitoring.

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-quota
  annotations:
    volume.kubernetes.io/selected-node: cn-zhangjiakou.192.168.XX.XX
  labels:
    app: web-quota
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: alicloud-local-quota
EOF

2.6 Deploy the application

Create a sample application

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web-quota
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: "nginx"
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
        - name: disk-ssd
          mountPath: /data
      volumes:
        - name: "disk-ssd"
          persistentVolumeClaim:
            claimName: csi-quota
EOF

After the pod starts, the /data directory in the container has 2 GiB of storage, matching the PVC request.

Verify step 2

Check that the PVC is bound and the pod is running:

kubectl get pvc
kubectl get pod

Expected output (LVM example):

NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
lvm-pvc   Bound    pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  2Gi        RWO            csi-local      1m

The PVC status must be Bound before proceeding. If it stays Pending, verify that the node name in the PVC annotation matches the value field in the ConfigMap.

Step 3: Automatically expand a local volume

Storage Auto Expander monitors PVC usage and triggers expansion based on a StorageAutoScalerPolicy. In this step, you install the plug-in and configure a policy that expands any PVC labeled app: web-quota by 50% when usage exceeds 80%, up to a maximum of 15 GiB.

3.1 Install the storage-auto-expander plug-in

For full installation instructions, see Use storage-operator to deploy and upgrade storage components.

  1. Edit the storage-operator ConfigMap to enable the plug-in:

    kubectl edit cm storage-operator -n kube-system

    Set the following:

      storage-auto-expander: |
    # deploy config
    install: true
    imageTag: "v1.18.8.1-d4301ee-aliyun"

  2. Verify the plug-in is running:

    kubectl get pod -n kube-system | grep storage-auto-expander

    Expected output:

    storage-auto-expander-6bb575b68c-tt4hh               1/1     Running     0          2m41s

3.2 Configure an auto-expansion policy

cat << EOF | kubectl apply -f -
apiVersion: storage.alibabacloud.com/v1alpha1
kind: StorageAutoScalerPolicy
metadata:
  name: hybrid-expand-policy
spec:
  pvcSelector:
    matchLabels:
      app: web-quota
  namespaces:
    - default
  conditions:
    - name: condition1
      key: volume-capacity-used-percentage
      operator: Gt
      values:
        - "80"
  actions:
    - name: action
      type: volume-expand
      params:
        scale: 50%
        limits: 15Gi
EOF

This policy targets PVCs with the label app: web-quota in the default namespace. When storage usage exceeds 80%, the volume expands by 50% of its current size. Expansion stops once the volume reaches 15 GiB.

What's next

Previous：Automatically collect JVM heap dumps on abnormal exits using CNFSNext：Prefill OSS data to high-performance volumes