您可以在本地盘上通过VolumeGroup进行磁盘虚拟化，并通过LVM数据卷切分磁盘给应用使用。本文介绍如何使用LVM数据卷。
背景信息
使用HostPath和LocalVolume都可以实现Pod对主机存储空间的访问，但均具有其局限性：
- Kubernetes没有提供上述本地存储卷的生命周期管理能力，需要管理员手动管理、运维存储卷。
- 多个Pod共享一个本地存储时，需要共享存储目录或者分别使用其子目录，无法做到容量隔离。
- 多个Pod共享一个本地存储时，IOPS、吞吐等指标共享了整个存储空间，无法进行限制。
- 新建Pod使用本地存储时，不了解各节点存储空间余量，无法进行合理的存储卷调度。
为此ACK提供了LVM数据卷方案，以解决上述问题。
功能介绍
- LVM数据卷生命周期管理：卷自动创建、删除、挂载、卸载。
- LVM数据卷扩容功能。
- LVM卷的监控能力。
- LVM卷的IOPS限制。
- 节点本地存储管理：自动运维VolumeGroup。
- LVM卷的集群容量感知能力。
注意事项
- LVM本地存储卷，不支持数据的跨节点迁移，不适合在高可用场景中使用。
- VolumeGroup本地存储容量感知为可选项（暂缓提供）。
实现架构
基础的LVM功能（卷的生命周期管理、扩容、挂载、格式化等）由CSI-Provisioner和CSI-Plugin实现。
| 单元 | 详情 |
|---|---|
| 存储管理器 | 节点本地存储（VolumeGroup）的运维管理以及容量统计。可以不用该组件，由Worker管理运维VolumeGroup。 |
| 存储信息中心 | 保存节点本地存储信息，包括容量、VolumeGroup等信息。 |
| 本地存储调度器 | 实现新建PVC对集群存储容量感知功能。 |
步骤一：为Plugin和Provisioner组件添加Secrets的RBAC权限
在部署Plugin和Provisioner组件前，需要先为clusterrole/alicloud-csi-plugin增加Secrets的RBAC权限。
步骤二：部署Plugin和Provisioner组件
LVM CSI插件分为2个组件：Plugin（负责挂载、卸载LVM卷）和Provisioner（负责创建LVM卷和PV对象）。
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: csi-plugin
namespace: kube-system
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
app: csi-plugin
template:
metadata:
creationTimestamp: null
labels:
app: csi-plugin
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: type
operator: NotIn
values:
- virtual-kubelet
containers:
- args:
- --v=5
- --csi-address=/var/lib/kubelet/csi-plugins/diskplugin.csi.alibabacloud.com/csi.sock
- --kubelet-registration-path=/var/lib/kubelet/csi-plugins/diskplugin.csi.alibabacloud.com/csi.sock
image: registry-vpc.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v1.3.0-6e9fff3-aliyun
imagePullPolicy: Always
name: disk-driver-registrar
resources:
limits:
cpu: 500m
memory: 1Gi
requests:
cpu: 10m
memory: 16Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/kubelet
name: kubelet-dir
- mountPath: /registration
name: registration-dir
- args:
- --v=5
- --csi-address=/var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/csi.sock
- --kubelet-registration-path=/var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/csi.sock
image: registry-vpc.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v1.3.0-6e9fff3-aliyun
imagePullPolicy: Always
name: nas-driver-registrar
resources:
limits:
cpu: 500m
memory: 1Gi
requests:
cpu: 10m
memory: 16Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/kubelet/
name: kubelet-dir
- mountPath: /registration
name: registration-dir
- args:
- --v=5
- --csi-address=/var/lib/kubelet/csi-plugins/ossplugin.csi.alibabacloud.com/csi.sock
- --kubelet-registration-path=/var/lib/kubelet/csi-plugins/ossplugin.csi.alibabacloud.com/csi.sock
image: registry-vpc.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v1.3.0-6e9fff3-aliyun
imagePullPolicy: Always
name: oss-driver-registrar
resources:
limits:
cpu: 500m
memory: 1Gi
requests:
cpu: 10m
memory: 16Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/kubelet/
name: kubelet-dir
- mountPath: /registration
name: registration-dir
- args:
- --endpoint=$(CSI_ENDPOINT)
- --v=2
- --driver=oss,nas,disk
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/csi-plugins/driverplugin.csi.alibabacloud.com-replace/csi.sock
- name: MAX_VOLUMES_PERNODE
value: "15"
- name: SERVICE_TYPE
value: plugin
image: registry-vpc.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.20.7-aafce42-aliyun
imagePullPolicy: Always
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: healthz
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 5
name: csi-plugin
ports:
- containerPort: 11260
hostPort: 11260
name: healthz
protocol: TCP
readinessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: healthz
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
cpu: 500m
memory: 1Gi
requests:
cpu: 100m
memory: 128Mi
securityContext:
allowPrivilegeEscalation: true
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/kubelet/
mountPropagation: Bidirectional
name: kubelet-dir
- mountPath: /host/etc
name: etc
- mountPath: /var/log/
name: host-log
- mountPath: /host/usr/
name: ossconnectordir
- mountPath: /var/lib/container
mountPropagation: Bidirectional
name: container-dir
- mountPath: /dev
mountPropagation: HostToContainer
name: host-dev
- mountPath: /var/addon
name: addon-token
readOnly: true
dnsPolicy: ClusterFirst
hostNetwork: true
hostPID: true
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-node-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: csi-admin
serviceAccountName: csi-admin
terminationGracePeriodSeconds: 30
tolerations:
- operator: Exists
volumes:
- hostPath:
path: /var/lib/kubelet/plugins_registry
type: DirectoryOrCreate
name: registration-dir
- hostPath:
path: /var/lib/container
type: DirectoryOrCreate
name: container-dir
- hostPath:
path: /var/lib/kubelet
type: Directory
name: kubelet-dir
- hostPath:
path: /dev
type: ""
name: host-dev
- hostPath:
path: /var/log/
type: ""
name: host-log
- hostPath:
path: /etc
type: ""
name: etc
- hostPath:
path: /usr/
type: ""
name: ossconnectordir
- name: addon-token
secret:
defaultMode: 420
items:
- key: addon.token.config
path: token-config
optional: true
secretName: addon.csi.token
updateStrategy:
rollingUpdate:
maxSurge: 0
maxUnavailable: 10%
type: RollingUpdate
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
name: localplugin.csi.alibabacloud.com
spec:
attachRequired: false
podInfoOnMount: true
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app: csi-local-plugin
name: csi-local-plugin
namespace: kube-system
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
app: csi-local-plugin
template:
metadata:
labels:
app: csi-local-plugin
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: type
operator: NotIn
values:
- virtual-kubelet
containers:
- args:
- '--v=5'
- '--csi-address=/csi/csi.sock'
- >-
--kubelet-registration-path=/var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com/csi.sock
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
image: >-
registry-vpc.{{ regionId }}.aliyuncs.com/acs/csi-node-driver-registrar:v2.3.1-038aeb6-aliyun
imagePullPolicy: Always
name: driver-registrar
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /csi
name: plugin-dir
- mountPath: /registration
name: registration-dir
- args:
- '--endpoint=$(CSI_ENDPOINT)'
- '--v=5'
- '--nodeid=$(KUBE_NODE_NAME)'
- '--driver=localplugin.csi.alibabacloud.com'
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: SERVICE_PORT
value: '11293'
- name: CSI_ENDPOINT
value: >-
unix://var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com/csi.sock
image: >-
registry-vpc.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.24.3-55228c1-aliyun
imagePullPolicy: Always
name: csi-localplugin
resources: {}
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- SYS_ADMIN
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/kubelet
mountPropagation: Bidirectional
name: pods-mount-dir
- mountPath: /dev
mountPropagation: HostToContainer
name: host-dev
- mountPath: /var/log/
name: host-log
- mountPath: /mnt
mountPropagation: Bidirectional
name: quota-path-dir
- mountPath: /tls/local/grpc
name: tls-token-dir
readOnly: true
dnsPolicy: ClusterFirst
hostNetwork: true
hostPID: true
priorityClassName: system-node-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: csi-admin
serviceAccountName: csi-admin
terminationGracePeriodSeconds: 30
tolerations:
- operator: Exists
volumes:
- name: tls-token-dir
secret:
defaultMode: 420
secretName: csi-local-plugin-cert
- hostPath:
path: /var/lib/kubelet/csi-plugins/localplugin.csi.alibabacloud.com
type: DirectoryOrCreate
name: plugin-dir
- hostPath:
path: /var/lib/kubelet/plugins_registry
type: DirectoryOrCreate
name: registration-dir
- hostPath:
path: /var/lib/kubelet
type: Directory
name: pods-mount-dir
- hostPath:
path: /dev
type: ''
name: host-dev
- hostPath:
path: /var/log/
type: ''
name: host-log
- hostPath:
path: /mnt
type: Directory
name: quota-path-dir
updateStrategy:
rollingUpdate:
maxUnavailable: 10%
type: RollingUpdate
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: csi-local-provisioner
name: csi-local-provisioner
namespace: kube-system
spec:
selector:
matchLabels:
app: csi-local-provisioner
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: csi-local-provisioner
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: node-role.kubernetes.io/master
operator: Exists
weight: 1
containers:
- args:
- --csi-address=$(ADDRESS)
- --feature-gates=Topology=True
- --volume-name-prefix=local
- --strict-topology=true
- --timeout=150s
- --extra-create-metadata=true
- --enable-leader-election=true
- --leader-election-type=leases
- --retry-interval-start=500ms
- --v=5
env:
- name: ADDRESS
value: /socketDir/csi.sock
image: registry.cn-hangzhou.aliyuncs.com/acs/csi-provisioner:v1.6.0-71838bd-aliyun
imagePullPolicy: Always
name: external-local-provisioner
volumeMounts:
- mountPath: /socketDir
name: socket-dir
- name: csi-localprovisioner
securityContext:
privileged: true
image: registry.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.20.7-aafce42-aliyun
imagePullPolicy: "Always"
args:
- "--endpoint=$(CSI_ENDPOINT)"
- "--v=2"
- "--driver=localplugin.csi.alibabacloud.com"
env:
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/csi-provisioner/localplugin.csi.alibabacloud.com/csi.sock
- name: SERVICE_TYPE
value: "provisioner"
- name: SERVICE_PORT
value: "11290"
volumeMounts:
- name: socket-dir
mountPath: /var/lib/kubelet/csi-provisioner/localplugin.csi.alibabacloud.com
- mountPath: /var/log/
name: host-log
- mountPath: /tls/local/grpc/
name: tls-token-dir
- args:
- --v=5
- --csi-address=$(ADDRESS)
- --leader-election
env:
- name: ADDRESS
value: /socketDir/csi.sock
image: registry.cn-hangzhou.aliyuncs.com/acs/csi-resizer:v1.1.0-7b30758-aliyun
imagePullPolicy: Always
name: external-local-resizer
volumeMounts:
- mountPath: /socketDir/
name: socket-dir
hostNetwork: true
serviceAccount: csi-admin
tolerations:
- effect: NoSchedule
operator: Exists
key: node-role.kubernetes.io/master
- effect: NoSchedule
operator: Exists
key: node.cloudprovider.kubernetes.io/uninitialized
volumes:
- name: socket-dir
emptyDir: {}
- name: tls-token-dir
emptyDir: {}
- hostPath:
path: /dev
type: ""
name: host-dev
- hostPath:
path: /var/log/
type: ""
name: host-log
- hostPath:
path: /mnt
type: Directory
name: quota-path-dir
- hostPath:
path: /var/lib/kubelet
type: Directory
name: pods-mount-dir
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: csi-local-provisioner
name: csi-local-provisioner
namespace: kube-system
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: csi-local-provisioner
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: csi-local-provisioner
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: node-role.kubernetes.io/master
operator: Exists
weight: 1
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: type
operator: NotIn
values:
- virtual-kubelet
containers:
- args:
- --csi-address=$(ADDRESS)
- --feature-gates=Topology=True
- --volume-name-prefix=local
- --strict-topology=true
- --timeout=150s
- --extra-create-metadata=true
- --enable-leader-election=true
- --leader-election-type=leases
- --retry-interval-start=500ms
- --default-fstype=ext4
- --v=5
env:
- name: ADDRESS
value: /socketDir/csi.sock
image: registry-vpc.cn-hangzhou.aliyuncs.com/acs/csi-provisioner:v3.0.0-080f01e64-aliyun
imagePullPolicy: Always
name: external-local-provisioner
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /socketDir
name: socket-dir
- args:
- --endpoint=$(CSI_ENDPOINT)
- --v=2
- --driver=localplugin.csi.alibabacloud.com
env:
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/csi-provisioner/localplugin.csi.alibabacloud.com/csi.sock
- name: SERVICE_TYPE
value: provisioner
- name: SERVICE_PORT
value: "11293"
image: registry-vpc.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.24.3-55228c1-aliyun
imagePullPolicy: Always
name: csi-localprovisioner
resources: {}
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/lib/kubelet/csi-provisioner/localplugin.csi.alibabacloud.com
name: socket-dir
- mountPath: /var/log/
name: host-log
- mountPath: /tls/local/grpc/
name: tls-token-dir
- args:
- --v=5
- --csi-address=$(ADDRESS)
- --leader-election
env:
- name: ADDRESS
value: /socketDir/csi.sock
image: registry-vpc.cn-hangzhou.aliyuncs.com/acs/csi-resizer:v1.3-ca84e84-aliyun
imagePullPolicy: Always
name: external-local-resizer
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /socketDir/
name: socket-dir
dnsPolicy: ClusterFirst
hostNetwork: true
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: csi-admin
serviceAccountName: csi-admin
terminationGracePeriodSeconds: 30
tolerations:
- operator: Exists
volumes:
- emptyDir: {}
name: socket-dir
- emptyDir: {}
name: tls-token-dir
- hostPath:
path: /dev
type: ""
name: host-dev
- hostPath:
path: /var/log/
type: ""
name: host-log
- hostPath:
path: /mnt
type: Directory
name: quota-path-dir
- hostPath:
path: /var/lib/kubelet
type: Directory
name: pods-mount-dir
步骤三：使用LVM存储卷
使用CSI-Provisioner自动创建PV，有以下特点：
- StorageClass中需要指定VolumeGroup名字。
- 如果期望创建的PV在某个节点，需要给PVC添加Annotations：volume.kubernetes.io/selected-node: nodeName。