您可以在本地盘上通过VolumeGroup进行磁盘虚拟化，并通过LVM数据卷切分磁盘给应用使用。本文介绍如何使用LVM数据卷。
背景信息
使用HostPath和LocalVolume都可以实现Pod对主机存储空间的访问，但均具有其局限性：
- Kubernetes没有提供上述本地存储卷的生命周期管理能力，需要管理员手动管理、运维存储卷。
- 多个Pod共享一个本地存储时，需要共享存储目录或者分别使用其子目录，无法做到容量隔离。
- 多个Pod共享一个本地存储时，IOPS、吞吐等指标共享了整个存储空间，无法进行限制。
- 新建Pod使用本地存储时，不了解各节点存储空间余量，无法进行合理的存储卷调度。
为此ACK提供了LVM数据卷方案，以解决上述问题。
功能介绍
- LVM数据卷生命周期管理：卷自动创建、删除、挂载、卸载。
- LVM数据卷扩容功能。
- LVM卷的监控能力。
- LVM卷的IOPS限制。
- 节点本地存储管理：自动运维VolumeGroup。
- LVM卷的集群容量感知能力。
注意事项
- LVM本地存储卷，不支持数据的跨节点迁移，不适合在高可用场景中使用。
- VolumeGroup本地存储容量感知为可选项（暂缓提供）。
实现架构
基础的LVM功能（卷的生命周期管理、扩容、挂载、格式化等）由CSI-Provisioner和CSI-Plugin实现。
| 单元 | 详情 |
|---|---|
| 存储管理器 | 节点本地存储（VolumeGroup）的运维管理以及容量统计。可以不用该组件，由Worker管理运维VolumeGroup。 |
| 存储信息中心 | 保存节点本地存储信息，包括容量、VolumeGroup等信息。 |
| 本地存储调度器 | 实现新建PVC对集群存储容量感知功能。 |
步骤一：为Plugin和Provisioner组件添加Secrets的RBAC权限
在部署Plugin和Provisioner组件前，需要先为clusterrole/alicloud-csi-plugin增加Secrets的RBAC权限。
- 执行以下命令，查看集群
clusterrole/alicloud-csi-plugin是否有Secrets的create权限。echo `JSONPATH='{range .rules[*]}{@.resources}:{@.verbs} \r\n {end}' \ && kubectl get clusterrole alicloud-csi-plugin -o jsonpath="$JSONPATH";` | grep secrets预期输出：
["secrets"]:["get","list"] - 若
clusterrole/alicloud-csi-plugin没有Secrets的create权限，请执行以下命令增加此权限。kubectl patch clusterrole alicloud-csi-plugin --type='json' -p='[{"op": "add", "path": "/rules/0", "value":{ "apiGroups": [""], "resources": ["secrets"], "verbs": ["create"]}}]'预期输出：
clusterrole.rbac.authorization.k8s.io/alicloud-csi-plugin patched - 执行以下命令，查看集群中
clusterrole/alicloud-csi-pluginSecrets的create权限是否添加成功。echo `JSONPATH='{range .rules[*]}{@.resources}:{@.verbs} \r\n {end}' \ && kubectl get clusterrole alicloud-csi-plugin -o jsonpath="$JSONPATH";` | grep secrets预期输出：
["secrets"]:["create"] ["secrets"]:["get","list"]从预期输出可得，集群中
clusterrole/alicloud-csi-pluginSecrets的create权限已添加成功。
步骤二：部署Plugin和Provisioner组件
LVM CSI插件分为2个组件：Plugin（负责挂载、卸载LVM卷）和Provisioner（负责创建LVM卷和PV对象）。
- 展开查看部署LVM CSI-Plugin（1.20集群及以下）
apiVersion: storage.k8s.io/v1beta1 kind: CSIDriver metadata: name: localplugin.csi.alibabacloud.com spec: attachRequired: false podInfoOnMount: true --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: csi-local-plugin name: csi-local-plugin namespace: kube-system spec: revisionHistoryLimit: 10 selector: matchLabels: app: csi-local-plugin template: metadata: labels: app: csi-local-plugin spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: type operator: NotIn values: - virtual-kubelet containers: - args: - '--v=5' - '--csi-address=/csi/csi.sock' - >- --kubelet-registration-path=/var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com/csi.sock env: - name: KUBE_NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName image: >- registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-node-driver-registrar:v1.3.0-6e9fff3-aliyun imagePullPolicy: Always name: driver-registrar resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /csi name: plugin-dir - mountPath: /registration name: registration-dir - args: - '--endpoint=$(CSI_ENDPOINT)' - '--v=5' - '--nodeid=$(KUBE_NODE_NAME)' - '--driver=localplugin.csi.alibabacloud.com' env: - name: KUBE_NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: SERVICE_PORT value: '11293' - name: CSI_ENDPOINT value: >- unix://var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com/csi.sock image: >- registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-plugin:v1.20.7-aafce42-aliyun imagePullPolicy: Always name: csi-localplugin resources: {} securityContext: allowPrivilegeEscalation: true capabilities: add: - SYS_ADMIN privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/lib/kubelet mountPropagation: Bidirectional name: pods-mount-dir - mountPath: /dev mountPropagation: HostToContainer name: host-dev - mountPath: /var/log/ name: host-log - mountPath: /mnt mountPropagation: Bidirectional name: quota-path-dir - mountPath: /tls/local/grpc name: tls-token-dir readOnly: true dnsPolicy: ClusterFirst hostNetwork: true hostPID: true priorityClassName: system-node-critical restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: csi-admin serviceAccountName: csi-admin terminationGracePeriodSeconds: 30 tolerations: - operator: Exists volumes: - name: tls-token-dir secret: defaultMode: 420 secretName: csi-local-plugin-cert - hostPath: path: /var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com type: DirectoryOrCreate name: plugin-dir - hostPath: path: /var/lib/kubelet/plugins_registry type: DirectoryOrCreate name: registration-dir - hostPath: path: /var/lib/kubelet type: Directory name: pods-mount-dir - hostPath: path: /dev type: '' name: host-dev - hostPath: path: /var/log/ type: '' name: host-log - hostPath: path: /mnt type: Directory name: quota-path-dir updateStrategy: rollingUpdate: maxUnavailable: 10% type: RollingUpdate - 展开查看部署LVM CSI-Provisioner（1.20集群及以下）
apiVersion: apps/v1 kind: Deployment metadata: labels: app: csi-local-provisioner name: csi-local-provisioner namespace: kube-system spec: selector: matchLabels: app: csi-local-provisioner strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app: csi-local-provisioner spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - preference: matchExpressions: - key: node-role.kubernetes.io/master operator: Exists weight: 1 containers: - args: - --csi-address=$(ADDRESS) - --feature-gates=Topology=True - --volume-name-prefix=local - --strict-topology=true - --timeout=150s - --extra-create-metadata=true - --enable-leader-election=true - --leader-election-type=leases - --retry-interval-start=500ms - --v=5 env: - name: ADDRESS value: /socketDir/csi.sock image: registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-provisioner:v1.6.0-71838bd-aliyun imagePullPolicy: Always name: external-local-provisioner volumeMounts: - mountPath: /socketDir name: socket-dir - name: csi-localprovisioner securityContext: privileged: true image: registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-plugin:v1.20.7-aafce42-aliyun imagePullPolicy: "Always" args: - "--endpoint=$(CSI_ENDPOINT)" - "--v=2" - "--driver=localplugin.csi.alibabacloud.com" env: - name: CSI_ENDPOINT value: unix://var/lib/kubelet/csi-provisioner/localplugin.csi.alibabacloud.com/csi.sock - name: SERVICE_TYPE value: "provisioner" - name: SERVICE_PORT value: "11290" volumeMounts: - name: socket-dir mountPath: /var/lib/kubelet/csi-provisioner/localplugin.csi.alibabacloud.com - mountPath: /var/log/ name: host-log - mountPath: /tls/local/grpc/ name: tls-token-dir - args: - --v=5 - --csi-address=$(ADDRESS) - --leader-election env: - name: ADDRESS value: /socketDir/csi.sock image: registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-resizer:v1.1.0-7b30758-aliyun imagePullPolicy: Always name: external-local-resizer volumeMounts: - mountPath: /socketDir/ name: socket-dir hostNetwork: true serviceAccount: csi-admin tolerations: - effect: NoSchedule operator: Exists key: node-role.kubernetes.io/master - effect: NoSchedule operator: Exists key: node.cloudprovider.kubernetes.io/uninitialized volumes: - name: socket-dir emptyDir: {} - name: tls-token-dir emptyDir: {} - hostPath: path: /dev type: "" name: host-dev - hostPath: path: /var/log/ type: "" name: host-log - hostPath: path: /mnt type: Directory name: quota-path-dir - hostPath: path: /var/lib/kubelet type: Directory name: pods-mount-dir - 展开查看部署LVM CSI-Plugin（1.22集群及以上）
apiVersion: storage.k8s.io/v1 kind: CSIDriver metadata: name: localplugin.csi.alibabacloud.com spec: attachRequired: false podInfoOnMount: true --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: csi-local-plugin name: csi-local-plugin namespace: kube-system spec: revisionHistoryLimit: 10 selector: matchLabels: app: csi-local-plugin template: metadata: labels: app: csi-local-plugin spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: type operator: NotIn values: - virtual-kubelet containers: - args: - '--v=5' - '--csi-address=/csi/csi.sock' - >- --kubelet-registration-path=/var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com/csi.sock env: - name: KUBE_NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName image: >- registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-node-driver-registrar:v2.3.1-038aeb6-aliyun imagePullPolicy: Always name: driver-registrar resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /csi name: plugin-dir - mountPath: /registration name: registration-dir - args: - '--endpoint=$(CSI_ENDPOINT)' - '--v=5' - '--nodeid=$(KUBE_NODE_NAME)' - '--driver=localplugin.csi.alibabacloud.com' env: - name: KUBE_NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: SERVICE_PORT value: '11293' - name: CSI_ENDPOINT value: >- unix://var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com/csi.sock image: >- registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-plugin:v1.24.3-55228c1-aliyun imagePullPolicy: Always name: csi-localplugin resources: {} securityContext: allowPrivilegeEscalation: true capabilities: add: - SYS_ADMIN privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/lib/kubelet mountPropagation: Bidirectional name: pods-mount-dir - mountPath: /dev mountPropagation: HostToContainer name: host-dev - mountPath: /var/log/ name: host-log - mountPath: /mnt mountPropagation: Bidirectional name: quota-path-dir - mountPath: /tls/local/grpc name: tls-token-dir readOnly: true dnsPolicy: ClusterFirst hostNetwork: true hostPID: true priorityClassName: system-node-critical restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: csi-admin serviceAccountName: csi-admin terminationGracePeriodSeconds: 30 tolerations: - operator: Exists volumes: - name: tls-token-dir secret: defaultMode: 420 secretName: csi-local-plugin-cert - hostPath: path: /var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com type: DirectoryOrCreate name: plugin-dir - hostPath: path: /var/lib/kubelet/plugins_registry type: DirectoryOrCreate name: registration-dir - hostPath: path: /var/lib/kubelet type: Directory name: pods-mount-dir - hostPath: path: /dev type: '' name: host-dev - hostPath: path: /var/log/ type: '' name: host-log - hostPath: path: /mnt type: Directory name: quota-path-dir updateStrategy: rollingUpdate: maxUnavailable: 10% type: RollingUpdate - 展开查看部署LVM CSI-Provisioner（1.22集群及以上）
apiVersion: apps/v1 kind: Deployment metadata: labels: app: csi-local-provisioner name: csi-local-provisioner namespace: kube-system spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: csi-local-provisioner strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: creationTimestamp: null labels: app: csi-local-provisioner spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - preference: matchExpressions: - key: node-role.kubernetes.io/master operator: Exists weight: 1 requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: type operator: NotIn values: - virtual-kubelet containers: - args: - --csi-address=$(ADDRESS) - --feature-gates=Topology=True - --volume-name-prefix=local - --strict-topology=true - --timeout=150s - --extra-create-metadata=true - --enable-leader-election=true - --leader-election-type=leases - --retry-interval-start=500ms - --default-fstype=ext4 - --v=5 env: - name: ADDRESS value: /socketDir/csi.sock image: registry-vpc.{{ regionId }}.com/acs/csi-provisioner:v3.0.0-080f01e64-aliyun imagePullPolicy: Always name: external-local-provisioner resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /socketDir name: socket-dir - args: - --endpoint=$(CSI_ENDPOINT) - --v=2 - --driver=localplugin.csi.alibabacloud.com env: - name: CSI_ENDPOINT value: unix://var/lib/kubelet/csi-provisioner/localplugin.csi.alibabacloud.com/csi.sock - name: SERVICE_TYPE value: provisioner - name: SERVICE_PORT value: "11293" image: registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-plugin:v1.24.3-55228c1-aliyun imagePullPolicy: Always name: csi-localprovisioner resources: {} securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/lib/kubelet/csi-provisioner/localplugin.csi.alibabacloud.com name: socket-dir - mountPath: /var/log/ name: host-log - mountPath: /tls/local/grpc/ name: tls-token-dir - args: - --v=5 - --csi-address=$(ADDRESS) - --leader-election env: - name: ADDRESS value: /socketDir/csi.sock image: registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-resizer:v1.3-ca84e84-aliyun imagePullPolicy: Always name: external-local-resizer resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /socketDir/ name: socket-dir dnsPolicy: ClusterFirst hostNetwork: true restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: csi-admin serviceAccountName: csi-admin terminationGracePeriodSeconds: 30 tolerations: - operator: Exists volumes: - emptyDir: {} name: socket-dir - emptyDir: {} name: tls-token-dir - hostPath: path: /dev type: "" name: host-dev - hostPath: path: /var/log/ type: "" name: host-log - hostPath: path: /mnt type: Directory name: quota-path-dir - hostPath: path: /var/lib/kubelet type: Directory name: pods-mount-dir
步骤三：使用LVM存储卷
使用CSI-Provisioner自动创建PV，有以下特点：
- StorageClass中需要指定VolumeGroup名字。
- 如果期望创建的PV在某个节点，需要给PVC添加Annotations：volume.kubernetes.io/selected-node: nodeName。
- 使用以下模板创建StorageClass。
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-local provisioner: localplugin.csi.alibabacloud.com parameters: volumeType: LVM vgName: volumegroup1 fsType: ext4 lvmType: "striping" writeIOPS: "10000" writeBPS: "1M" reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true参数 描述 volumeType 表示本地存储类型为LVM（后续将支持其他类型本地存储）。 vgName 必选，VolumeGroup的名字。 fsType 格式文件系统类型。 lvmType 生成的LVM类型，支持linear（线性）、striping（条带化）。 writeIOPS 使用StorageClass动态创建每一个LV时，限制每一个LV每秒写IO的次数。 writeBPS 使用StorageClass动态创建每一个LV时，限制每一个LV每秒写IO的大小，单位为Byte。 - 使用以下模板创建PVC。
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: lvm-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi storageClassName: csi-local - 使用以下模板创建应用。
apiVersion: apps/v1 kind: Deployment metadata: name: deployment-lvm labels: app: nginx spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 volumeMounts: - name: lvm-pvc mountPath: "/data" volumes: - name: lvm-pvc persistentVolumeClaim: claimName: lvm-pvc - 查看应用状态。
- 扩容LVM卷。
- 执行以下命令监控LVM卷。
curl -s localhost:10255/metrics | grep lvm-pvc预期输出：kubelet_volume_stats_available_bytes{namespace="default",persistentvolumeclaim="lvm-pvc"} 1.917165568e+09 kubelet_volume_stats_capacity_bytes{namespace="default",persistentvolumeclaim="lvm-pvc"} 1.939816448e+09 kubelet_volume_stats_inodes{namespace="default",persistentvolumeclaim="lvm-pvc"} 122400 kubelet_volume_stats_inodes_free{namespace="default",persistentvolumeclaim="lvm-pvc"} 122389 kubelet_volume_stats_inodes_used{namespace="default",persistentvolumeclaim="lvm-pvc"} 11 kubelet_volume_stats_used_bytes{namespace="default",persistentvolumeclaim="lvm-pvc"} 5.873664e+06可以将上述监控数据接入Prometheus并在前端展示。具体操作，请参见开源Prometheus监控。