Use LVM local storage in an ACK cluster-阿里云帮助中心

You can use a VolumeGroup to virtualize local disks and partition them into Logical Volume Manager (LVM) volumes for your applications. This topic describes how to use LVM volumes.

Background

Both HostPath and LocalVolume allow pods to access host storage, but they have limitations. To address these limitations, ACK provides an LVM volume solution. The main limitations of HostPath and LocalVolume are listed below:

Kubernetes does not manage the lifecycle of these local storage volumes. An administrator must manually manage and perform operations and maintenance (O&M) on the storage volumes.
When multiple pods share local storage, they must either share a storage directory or use separate subdirectories. This prevents capacity isolation.
When multiple pods share local storage, metrics such as input/output operations per second (IOPS) and throughput apply to the entire storage space and cannot be configured for individual pods.
When a new pod uses local storage, the scheduler is unaware of the remaining storage space on each node. This can lead to inefficient volume scheduling.

Features

LVM volume lifecycle management: Automatic creation, deletion, mounting, and unmounting of volumes.
LVM volume scale-out.
LVM volume monitoring.
IOPS limits for LVM volumes.
Node local storage management: Automatic O&M for VolumeGroups.
Cluster capacity awareness for LVM volumes.

Notes

LVM local storage volumes do not support data migration across nodes. They are not suitable for high availability (HA) scenarios.
Capacity awareness for VolumeGroup local storage is an optional feature. The initialization of a VolumeGroup depends on local storage resources. You must initialize the VolumeGroup manually or automatically initialize local storage resources. Both methods require a deep understanding of local storage resources. If you are not familiar with local storage resources, use cloud storage resources such as disks or Container Network File System (CNFS) instead.

Architecture

The CSI-Provisioner and CSI-Plugin components implement basic LVM features, such as volume lifecycle management, scale-out, mounting, and formatting.

Component	Details
Storage manager	Manages O&M and capacity statistics for node local storage. This component is optional. The Worker can manage O&M for the VolumeGroup.
Storage information center	Saves node local storage information, including capacity and VolumeGroup details.
Local storage scheduler	Enables cluster storage capacity awareness for new PersistentVolumeClaims (PVCs).

Step 1: Create a ServiceAccount for the plugin and provisioner components

For clusters that run Kubernetes 1.22 or later, you must create a new ServiceAccount. For clusters that run Kubernetes 1.20 or earlier, you must add Role-Based Access Control (RBAC) permissions for Secrets to clusterrole/alicloud-csi-plugin.

Clusters of version 1.22 and later

Create a ServiceAccount and grant it permissions using the following YAML file:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: alibaba-cloud-csi-local
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: alibaba-cloud-csi-local
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update", "create", "delete", "patch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims/status"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "update", "patch", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "create", "list", "watch", "delete", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments/status"]
    verbs: ["patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: alibaba-cloud-csi-local
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: alibaba-cloud-csi-local
subjects:
- kind: ServiceAccount
  name: alibaba-cloud-csi-local
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: alibaba-cloud-csi-local
  namespace: kube-system
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["csi-local-plugin-cert"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["csi-plugin", "ack-cluster-profile"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: alibaba-cloud-csi-local
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: alibaba-cloud-csi-local
subjects:
- kind: ServiceAccount
  name: alibaba-cloud-csi-local
  namespace: kube-system

Clusters of version 1.20 and earlier

The csi-plugin component shares the same ServiceAccount. First, add RBAC permissions for Secrets to clusterrole/alicloud-csi-plugin.

Check whether clusterrole/alicloud-csi-plugin has the create permission for Secrets.

echo `JSONPATH='{range .rules[*]}{@.resources}:{@.verbs} \r\n
{end}' \
 && kubectl get clusterrole  alicloud-csi-plugin -o jsonpath="$JSONPATH";` | grep secrets

Expected output:

["secrets"]:["get","list"]

If clusterrole/alicloud-csi-plugin does not have the create permission for Secrets, run the following command to add this permission.

 kubectl patch clusterrole alicloud-csi-plugin --type='json' -p='[{"op": "add", "path": "/rules/0", "value":{ "apiGroups": [""], "resources": ["secrets"], "verbs": ["create"]}}]'

Expected output:

clusterrole.rbac.authorization.k8s.io/alicloud-csi-plugin patched

Verify that the create permission for Secrets is added to clusterrole/alicloud-csi-plugin.
```
echo `JSONPATH='{range .rules[*]}{@.resources}:{@.verbs} \r\n
{end}' \
 && kubectl get clusterrole  alicloud-csi-plugin -o jsonpath="$JSONPATH";` | grep secrets
```
Expected output:
```
["secrets"]:["create"] 
["secrets"]:["get","list"]
```
The output indicates that the create permission for Secrets is successfully added to clusterrole/alicloud-csi-plugin.

Step 2: Deploy the plugin and provisioner components

The Container Storage Interface (CSI) solution consists of two components: the plugin, which mounts and unmounts volumes, and the provisioner, which creates volumes and persistent volume (PV) objects.

Note

Replace {{ regionId }} in the following YAML files with the ID of the region where your cluster is located.

Clusters of version 1.22 and later

Deploy the CSI plugin

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: localplugin.csi.alibabacloud.com
spec:
  attachRequired: false
  podInfoOnMount: true
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: csi-local-plugin
  name: csi-local-plugin
  namespace: kube-system
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: csi-local-plugin
  template:
    metadata:
      labels:
        app: csi-local-plugin
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: type
                    operator: NotIn
                    values:
                      - virtual-kubelet
      containers:
        - args:
            - '--v=5'
            - '--csi-address=/csi/csi.sock'
            - >-
              --kubelet-registration-path=/var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com/csi.sock
          env:
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
          image: >-
            registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-node-driver-registrar:v2.3.1-038aeb6-aliyun
          imagePullPolicy: Always
          name: driver-registrar
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /csi
              name: plugin-dir
            - mountPath: /registration
              name: registration-dir
        - args:
            - '--endpoint=$(CSI_ENDPOINT)'
            - '--v=5'
            - '--nodeid=$(KUBE_NODE_NAME)'
            - '--driver=localplugin.csi.alibabacloud.com'
          env:
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: SERVICE_PORT
              value: '11293'
            - name: CSI_ENDPOINT
              value: >-
                unix://var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com/csi.sock
          image: >-
             registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-plugin:v1.24.3-55228c1-aliyun
          imagePullPolicy: Always
          name: csi-localplugin
          resources: {}
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              add:
                - SYS_ADMIN
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /var/lib/kubelet
              mountPropagation: Bidirectional
              name: pods-mount-dir
            - mountPath: /dev
              mountPropagation: HostToContainer
              name: host-dev
            - mountPath: /var/log/
              name: host-log
            - mountPath: /mnt
              mountPropagation: Bidirectional
              name: quota-path-dir
            - mountPath: /tls/local/grpc
              name: tls-token-dir
              readOnly: true
      dnsPolicy: ClusterFirst
      hostNetwork: true
      hostPID: true
      priorityClassName: system-node-critical
      restartPolicy: Always
      securityContext: {}
      serviceAccountName: alibaba-cloud-csi-local
      terminationGracePeriodSeconds: 30
      tolerations:
        - operator: Exists
      volumes:
        - name: tls-token-dir
          secret:
            defaultMode: 420
            secretName: csi-local-plugin-cert
        - hostPath:
            path: /var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com
            type: DirectoryOrCreate
          name: plugin-dir
        - hostPath:
            path: /var/lib/kubelet/plugins_registry
            type: DirectoryOrCreate
          name: registration-dir
        - hostPath:
            path: /var/lib/kubelet
            type: Directory
          name: pods-mount-dir
        - hostPath:
            path: /dev
            type: ''
          name: host-dev
        - hostPath:
            path: /var/log/
            type: ''
          name: host-log
        - hostPath:
            path: /mnt
            type: Directory
          name: quota-path-dir
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%
    type: RollingUpdate

Deploy the CSI provisioner

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: csi-local-provisioner
  name: csi-local-provisioner
  namespace: kube-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: csi-local-provisioner
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: csi-local-provisioner
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - preference:
              matchExpressions:
              - key: node-role.kubernetes.io/master
                operator: Exists
            weight: 1
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: type
                operator: NotIn
                values:
                - virtual-kubelet
      containers:
      - args:
        - --csi-address=$(ADDRESS)
        - --feature-gates=Topology=True
        - --volume-name-prefix=local
        - --strict-topology=true
        - --timeout=150s
        - --extra-create-metadata=true
        - --enable-leader-election=true
        - --leader-election-type=leases
        - --retry-interval-start=500ms
        - --default-fstype=ext4
        - --v=5
        env:
        - name: ADDRESS
          value: /socketDir/csi.sock
        image: registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-provisioner:v3.0.0-080f01e64-aliyun
        imagePullPolicy: Always
        name: external-local-provisioner
        resources: {}
        volumeMounts:
        - mountPath: /socketDir
          name: socket-dir
      - args:
        - --endpoint=$(CSI_ENDPOINT)
        - --v=2
        - --driver=localplugin.csi.alibabacloud.com
        env:
        - name: CSI_ENDPOINT
          value: unix://var/lib/kubelet/csi-provisioner/localplugin.csi.alibabacloud.com/csi.sock
        - name: SERVICE_TYPE
          value: provisioner
        - name: SERVICE_PORT
          value: "11290"
        image: registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-plugin:v1.24.3-55228c1-aliyun
        imagePullPolicy: Always
        name: csi-localprovisioner
        resources: {}
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /var/lib/kubelet/csi-provisioner/localplugin.csi.alibabacloud.com
          name: socket-dir
        - mountPath: /var/log/
          name: host-log
        - mountPath: /tls/local/grpc/
          name: tls-token-dir
      - args:
        - --v=5
        - --csi-address=$(ADDRESS)
        - --leader-election
        env:
        - name: ADDRESS
          value: /socketDir/csi.sock
        image: registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-resizer:v1.3-ca84e84-aliyun
        imagePullPolicy: Always
        name: external-local-resizer
        resources: {}
        volumeMounts:
        - mountPath: /socketDir/
          name: socket-dir
      dnsPolicy: ClusterFirst
      hostNetwork: true
      restartPolicy: Always
      securityContext: {}
      serviceAccountName: alibaba-cloud-csi-local
      terminationGracePeriodSeconds: 30
      tolerations:
      - operator: Exists
      volumes:
      - emptyDir: {}
        name: socket-dir
      - emptyDir: {}
        name: tls-token-dir
      - hostPath:
          path: /dev
          type: ""
        name: host-dev
      - hostPath:
          path: /var/log/
          type: ""
        name: host-log
      - hostPath:
          path: /mnt
          type: Directory
        name: quota-path-dir
      - hostPath:
          path: /var/lib/kubelet
          type: Directory
        name: pods-mount-dir

Clusters of version 1.20 and earlier

Deploy the CSI plugin

apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
  name: localplugin.csi.alibabacloud.com
spec:
  attachRequired: false
  podInfoOnMount: true
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: csi-local-plugin
  name: csi-local-plugin
  namespace: kube-system
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: csi-local-plugin
  template:
    metadata:
      labels:
        app: csi-local-plugin
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: type
                    operator: NotIn
                    values:
                      - virtual-kubelet
      containers:
        - args:
            - '--v=5'
            - '--csi-address=/csi/csi.sock'
            - >-
              --kubelet-registration-path=/var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com/csi.sock
          env:
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
          image: >-
            registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-node-driver-registrar:v1.3.0-6e9fff3-aliyun
          imagePullPolicy: Always
          name: driver-registrar
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /csi
              name: plugin-dir
            - mountPath: /registration
              name: registration-dir
        - args:
            - '--endpoint=$(CSI_ENDPOINT)'
            - '--v=5'
            - '--nodeid=$(KUBE_NODE_NAME)'
            - '--driver=localplugin.csi.alibabacloud.com'
          env:
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: SERVICE_PORT
              value: '11293'
            - name: CSI_ENDPOINT
              value: >-
                unix://var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com/csi.sock
          image: >-
            registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-plugin:v1.20.7-aafce42-aliyun
          imagePullPolicy: Always
          name: csi-localplugin
          resources: {}
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              add:
                - SYS_ADMIN
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /var/lib/kubelet
              mountPropagation: Bidirectional
              name: pods-mount-dir
            - mountPath: /dev
              mountPropagation: HostToContainer
              name: host-dev
            - mountPath: /var/log/
              name: host-log
            - mountPath: /mnt
              mountPropagation: Bidirectional
              name: quota-path-dir
            - mountPath: /tls/local/grpc
              name: tls-token-dir
              readOnly: true
      dnsPolicy: ClusterFirst
      hostNetwork: true
      hostPID: true
      priorityClassName: system-node-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: csi-admin
      serviceAccountName: csi-admin
      terminationGracePeriodSeconds: 30
      tolerations:
        - operator: Exists
      volumes:
        - name: tls-token-dir
          secret:
            defaultMode: 420
            secretName: csi-local-plugin-cert
        - hostPath:
            path: /var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com
            type: DirectoryOrCreate
          name: plugin-dir
        - hostPath:
            path: /var/lib/kubelet/plugins_registry
            type: DirectoryOrCreate
          name: registration-dir
        - hostPath:
            path: /var/lib/kubelet
            type: Directory
          name: pods-mount-dir
        - hostPath:
            path: /dev
            type: ''
          name: host-dev
        - hostPath:
            path: /var/log/
            type: ''
          name: host-log
        - hostPath:
            path: /mnt
            type: Directory
          name: quota-path-dir
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%
    type: RollingUpdate

Deploy the CSI provisioner

apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
   app: csi-local-provisioner
 name: csi-local-provisioner
 namespace: kube-system
spec:
 selector:
   matchLabels:
     app: csi-local-provisioner
 strategy:
   rollingUpdate:
     maxSurge: 25%
     maxUnavailable: 25%
   type: RollingUpdate
 template:
   metadata:
     labels:
       app: csi-local-provisioner
   spec:
     affinity:
       nodeAffinity:
         preferredDuringSchedulingIgnoredDuringExecution:
         - preference:
             matchExpressions:
             - key: node-role.kubernetes.io/master
               operator: Exists
           weight: 1
     containers:
     - args:
       - --csi-address=$(ADDRESS)
       - --feature-gates=Topology=True
       - --volume-name-prefix=local
       - --strict-topology=true
       - --timeout=150s
       - --extra-create-metadata=true
       - --enable-leader-election=true
       - --leader-election-type=leases
       - --retry-interval-start=500ms
       - --v=5
       env:
       - name: ADDRESS
         value: /socketDir/csi.sock
       image: registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-provisioner:v1.6.0-71838bd-aliyun
       imagePullPolicy: Always
       name: external-local-provisioner
       volumeMounts:
       - mountPath: /socketDir
         name: socket-dir
     - name: csi-localprovisioner
       securityContext:
         privileged: true
       image: registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-plugin:v1.20.7-aafce42-aliyun
       imagePullPolicy: "Always"
       args:
         - "--endpoint=$(CSI_ENDPOINT)"
         - "--v=2"
         - "--driver=localplugin.csi.alibabacloud.com"
       env:
       - name: CSI_ENDPOINT
         value: unix://var/lib/kubelet/csi-provisioner/localplugin.csi.alibabacloud.com/csi.sock
       - name: SERVICE_TYPE
         value: "provisioner"
       - name: SERVICE_PORT
         value: "11290"
       volumeMounts:
       - name: socket-dir
         mountPath: /var/lib/kubelet/csi-provisioner/localplugin.csi.alibabacloud.com
       - mountPath: /var/log/
         name: host-log
       - mountPath: /tls/local/grpc/
         name: tls-token-dir
     - args:
       - --v=5
       - --csi-address=$(ADDRESS)
       - --leader-election
       env:
       - name: ADDRESS
         value: /socketDir/csi.sock
       image: registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-resizer:v1.1.0-7b30758-aliyun
       imagePullPolicy: Always
       name: external-local-resizer
       volumeMounts:
       - mountPath: /socketDir/
         name: socket-dir
     hostNetwork: true
     serviceAccount: csi-admin
     tolerations:
     - effect: NoSchedule
       operator: Exists
       key: node-role.kubernetes.io/master
     - effect: NoSchedule
       operator: Exists
       key: node.cloudprovider.kubernetes.io/uninitialized
     volumes:
     - name: socket-dir
       emptyDir: {}
     - name: tls-token-dir
       emptyDir: {}
     - hostPath:
         path: /dev
         type: ""
       name: host-dev
     - hostPath:
         path: /var/log/
         type: ""
       name: host-log
     - hostPath:
         path: /mnt
         type: Directory
       name: quota-path-dir
     - hostPath:
         path: /var/lib/kubelet
         type: Directory
       name: pods-mount-dir

Step 3: Use an LVM storage volume

When you use the Container Storage Interface (CSI) provisioner to automatically create a PersistentVolume (PV), note the following:

The StorageClass must specify the VolumeGroup name.
If you want the PV to be created on a specific node, add the annotation volume.kubernetes.io/selected-node: nodeName to the PersistentVolumeClaim (PVC).

Create a StorageClass using the following YAML file.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
    name: csi-local
provisioner: localplugin.csi.alibabacloud.com
parameters:
    volumeType: LVM
    vgName: volumegroup1
    fsType: ext4
    lvmType: "striping"
    writeIOPS: "10000"
    writeBPS: "1M"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Parameter	Description
volumeType	Specifies that the local storage type is LVM.
vgName	Required. The name of the VolumeGroup.
fsType	The file system type to be formatted.
lvmType	The type of LVM to create. Valid values are `linear` and `striping`.
writeIOPS	When you dynamically create a logical volume (LV) using the StorageClass, this parameter limits the number of write IOPS for each LV.
writeBPS	When you dynamically create an LV using the StorageClass, this parameter limits the write BPS for each LV. The unit is bytes.

Create a PVC using the following YAML file.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lvm-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: csi-local

Create an application using the following YAML file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-lvm
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
        volumeMounts:
          - name: lvm-pvc
            mountPath: "/data"
      volumes:
        - name: lvm-pvc
          persistentVolumeClaim:
            claimName: lvm-pvc

Check the PVC and PV information.

Check the PVC information.

kubectl get pvc

Expected output:

NAME      STATUS   VOLUME                                      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
lvm-pvc   Bound    disk-afacf7a9-3d1a-45da-b443-24f8fb35****   2Gi        RWO            csi-local      16s

Check the PV information.

kubectl get pv

Expected output:

NAME                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS   REASON   AGE
disk-afacf7a9-3d1a-45da-b443-24f8fb35****   2Gi        RWO            Delete           Bound    default/lvm-pvc   csi-local               12s

Check the application status.

Check the pod information.

kubectl get pod

Expected output:

NAME                             READY   STATUS    RESTARTS   AGE
deployment-lvm-9f798687c-m****   1/1     Running   0          9s

Check the mount details of the pod.

kubectl exec -ti deployment-lvm-9f798687c-m**** -- df /data

Expected output:

Filesystem                                                                1K-blocks  Used   Available Use% Mounted on
/dev/mapper/volumegroup1-disk--afacf7a9--3d1a--45da--b443--24f8fb35****   1998672    6144   1976144   1%   /data

List the directories under /data.
```
ls /data
```
Expected output:
```
lost+found
```

Verify the persistent storage of the LVM volume.

Create a `test` directory in /data and then list the contents of the directory.
```
touch /data/test
ls /data
```
Expected output:
```
lost+found  test
```
Log out of the pod.
```
exit
```

Delete the pod. The system automatically recreates the pod.

kubectl delete pod deployment-lvm-9f798687c-m****

Expected output:

pod "deployment-lvm-9f798687c-m****" deleted

Check the pod information.

kubectl get pod

Expected output:

NAME                             READY   STATUS    RESTARTS   AGE
deployment-lvm-9f798687c-j****   1/1     Running   0          2m19s

Check the mount details of the pod.

kubectl exec deployment-lvm-9f798687c-j**** -- ls /data

Expected output:

lost+found  
test

Expand the LVM volume.

Check the PVC information.

kubectl get pvc

Expected output:

NAME      STATUS   VOLUME                                      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
lvm-pvc   Bound    disk-afacf7a9-3d1a-45da-b443-24f8fb35****   2Gi        RWO            csi-local      6m50s

Expand the PVC to 4 GiB.

kubectl patch pvc lvm-pvc -p '{"spec":{"resources":{"requests":{"storage":"4Gi"}}}}'

Expected output:

persistentvolumeclaim/lvm-pvc patched

Check the PVC information.

kubectl get pvc

Expected output:

NAME      STATUS   VOLUME                                      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
lvm-pvc   Bound    disk-afacf7a9-3d1a-45da-b443-24f8fb35****   4Gi        RWO            csi-local      7m26s

Verify that the LVM volume is expanded to 4 GiB.

kubectl exec deployment-lvm-9f798687c-j**** -- df /data

Expected output:

Filesystem                                                                1K-blocks  Used   Available Use% Mounted on
/dev/mapper/volumegroup1-disk--afacf7a9--3d1a--45da--b443--24f8fb35****   4062912    8184   4038344   1%   /data

Monitor the LVM volume.

curl -s localhost:10255/metrics | grep lvm-pvc

Expected output:

kubelet_volume_stats_available_bytes{namespace="default",persistentvolumeclaim="lvm-pvc"} 1.917165568e+09
kubelet_volume_stats_capacity_bytes{namespace="default",persistentvolumeclaim="lvm-pvc"} 1.939816448e+09
kubelet_volume_stats_inodes{namespace="default",persistentvolumeclaim="lvm-pvc"} 122400
kubelet_volume_stats_inodes_free{namespace="default",persistentvolumeclaim="lvm-pvc"} 122389
kubelet_volume_stats_inodes_used{namespace="default",persistentvolumeclaim="lvm-pvc"} 11
kubelet_volume_stats_used_bytes{namespace="default",persistentvolumeclaim="lvm-pvc"} 5.873664e+06

You can ingest this monitoring data into Prometheus and display it on the frontend. For more information, see Open source Prometheus monitoring.