文档

自建Kubernetes挂载NAS

更新时间:

本文介绍如何将NAS挂载至自建的Kubernetes集群。

插件选择

NAS支持通过阿里云CSI存储组件、通过Kubernetes内置的Volume或通过CSI插件将NAS挂载至自建Kubernetes集群访问。请根据实际业务,选择合适的插件挂载NAS。

插件

说明

阿里云CSI存储组件(推荐)

阿里云ACK深度融合阿里云存储服务,完全兼容Kubernetes原生的存储服务,可通过部署CSI存储组件接入阿里云存储服务。结合ACK One注册集群您可以直接将云下Kubernetes集群接入云端,快速搭建混合云集群,可以将本地数据中心Kubernetes集群或其他云厂商Kubernetes集群接入阿里云容器服务管理平台,进行统一管理。

ACK支持Pod自动绑定阿里云云盘、NAS、OSS、CPFS、本地卷等存储服务。具体操作,请参见:通过阿里云CSI存储组件挂载NAS(推荐)

Volume

Volume为Kubernetes内置存储卷,具备良好的兼容性,操作步骤简单。如果您不想使用阿里云CSI插件,推荐您使用Volume方式使用阿里云NAS文件系统。更多信息,请参见方法一:通过Volume方式挂载

CSI插件

CSI插件为非阿里云CSI组件,通常适用于初始状态下的Kubernetes。具体操作,请参见方法二:通过CSI插件挂载

重要

阿里云不会对该插件的版本、安全性、准确性进行任何保证,在使用过程中阿里云不对由此引发的任何损害承担责任。请根据实际业务,选择合适的插件挂载NAS。

通过阿里云CSI存储组件挂载NAS(推荐)

前提条件

注意事项

  • 如果您的自建Kubernetes集群部署在阿里云ECS上,您需要为ECS实例添加节点标签。关于如何为ECS实例添加节点标签,请参见自建Kubernetes集群接入ACK注册集群后的ECS节点初始化配置

  • 如果您使用注册集群节点池功能为本地数据中心自建Kubernetes集群扩容的阿里云ECS节点,则默认已添加节点标签alibabacloud.com/external=true

步骤一:为CSI组件配置RAM权限

通过onectl配置

  1. 在本地安装配置onectl。具体操作,请参见通过onectl管理注册集群

  2. onectl通过RAM用户的AccessKey来访问云服务资源,您可以执行以下命令,配置RAM用户的CSI组件权限。

    onectl ram-user grant --addon csi-plugin

    预期输出:

    Ram policy ack-one-registered-cluster-policy-csi-plugin granted to ram user ack-one-user-ce313528c3 successfully.

通过控制台配置

在注册集群中安装CSI存储组件前,您需要在自建Kubernetes集群中设置AccessKey用来访问云服务的权限。设置AccessKey前,您需要创建RAM用户并为其添加访问相关云资源的权限。

  1. 创建RAM用户。具体操作,请参见创建RAM用户

  2. 创建自定义权限策略。

    关于创建自定义权限策略的操作步骤,请参见创建自定义权限策略

    通过以下自定义策略示例,添加磁盘、快照、快照策略、资源标签、实例、文件系统及仓库的管理权限。关于API参考的更多信息,请参见API概览

    展开查看自定义策略示例内容

    {
        "Version": "1",
        "Statement": [
            {
                "Action": [
                    "ecs:AttachDisk",
                    "ecs:DetachDisk",
                    "ecs:DescribeDisks",
                    "ecs:CreateDisk",
                    "ecs:ResizeDisk",
                    "ecs:CreateSnapshot",
                    "ecs:DeleteSnapshot",
                    "ecs:CreateAutoSnapshotPolicy",
                    "ecs:ApplyAutoSnapshotPolicy",
                    "ecs:CancelAutoSnapshotPolicy",
                    "ecs:DeleteAutoSnapshotPolicy",
                    "ecs:DescribeAutoSnapshotPolicyEX",
                    "ecs:ModifyAutoSnapshotPolicyEx",
                    "ecs:AddTags",
                    "ecs:DescribeTags",
                    "ecs:DescribeSnapshots",
                    "ecs:ListTagResources",
                    "ecs:TagResources",
                    "ecs:UntagResources",
                    "ecs:ModifyDiskSpec",
                    "ecs:CreateSnapshot",
                    "ecs:DeleteDisk",
                    "ecs:DescribeInstanceAttribute",
                    "ecs:DescribeInstances"
                ],
                "Resource": [
                    "*"
                ],
                "Effect": "Allow"
            },
            {
                "Action": [
                    "nas:DescribeFileSystems",
                    "nas:DescribeMountTargets",
                    "nas:AddTags",
                    "nas:DescribeTags",
                    "nas:RemoveTags",
                    "nas:CreateFileSystem",
                    "nas:DeleteFileSystem",
                    "nas:ModifyFileSystem",
                    "nas:CreateMountTarget",
                    "nas:DeleteMountTarget",
                    "nas:ModifyMountTarget",
                    "nas:TagResources",
                    "nas:SetDirQuota",
                    "nas:EnableRecycleBin",
                    "nas:GetRecycleBinAttribute"
                ],
                "Resource": [
                    "*"
                ],
                "Effect": "Allow"
            },
            {
                "Action": [
                    "oss:PutBucket",
                    "oss:GetObjectTagging",
                    "oss:ListBuckets",
                    "oss:PutBucketTags",
                    "oss:GetBucketTags",
                    "oss:PutBucketEncryption",
                    "oss:GetBucketInfo"
                ],
                "Resource": [
                    "*"
                ],
                "Effect": "Allow"
            }
        ]
    }
  3. 为RAM用户添加权限。具体操作,请参见为RAM用户授权

  4. 为RAM用户创建AccessKey。具体操作,请参见获取AccessKey

  5. 使用AccessKey在注册集群中创建名为alibaba-addon-secret的Secret资源。

    安装CSI组件时将自动引用此AccessKey访问对应的云服务资源。

    kubectl -n kube-system create secret generic alibaba-addon-secret --from-literal='access-key-id=<your access key id>' --from-literal='access-key-secret=<your access key secret>'
    说明

    <your access key id><your access key secret>为上一步获取的AccessKey信息。

步骤二:安装CSI插件

通过onectl安装

执行以下命令,安装CSI组件。

onectl addon install csi-plugin
onectl addon install csi-provisioner

预期输出:

Addon csi-plugin, version **** installed.
Addon csi-provisioner, version **** installed.

通过控制台安装

  1. 登录容器服务管理控制台,在左侧导航栏选择集群

  2. 集群列表页面,单击目标集群名称,然后在左侧导航栏,选择运维管理 > 组件管理

  3. 单击存储页签,在csi-plugincsi-provisioner卡片单击安装

  4. 提示对话框中确认版本信息后,单击确定

步骤三:使用存储卷

您可以通过以下两种方式挂载NAS文件系统,持久化应用数据。

自建Kubernetes直接挂载NAS

对于在物理机中自建的Kubernetes集群,您还可以通过以下两种方式挂载NAS并进行访问。推荐您优先选择Volume方式使用阿里云NAS文件系统,相比CSI插件方式,Volume为Kubernetes内置存储卷,具备更好的兼容性,操作步骤更简单。

前提条件

  • 已创建NFS协议文件系统。具体操作,请参见创建文件系统

    若需要加密NAS文件系统中的数据,请在创建NAS文件系统时配置加密类型。

  • 已添加挂载点。具体操作,请参见添加挂载点

  • 已自建Kubernetes集群且集群版本为1.16及以上版本。

  • 已确认Pod调度节点的安全组规则开放111端口和2049端口。具体操作,请参见添加安全组规则

  • 如果您的Kubernetes集群部署在线下IDC,请确保已打通IDC与阿里云的网络。您可通过阿里云高速通道、智能接入网关或者VPN网关打通IDC和云上网络。相关参考如下:

使用限制

NAS文件系统在挂载连通性、文件系统数量及协议类型等方面存在相应的约束条件。详细信息,请参见使用限制

重要

在没有卸载NAS文件系统前,务必不要删除NAS挂载点,否则会造成操作系统无响应。

方法一:通过Volume方式挂载

  1. 登录自建Kubernetes集群节点。

  2. 配置Kubernetes节点。

    您需要为Kubernetes集群中的每个节点都进行配置。如果该集群有新增节点时,您也需要为其配置以下操作然后再将新增节点加入到Kubernetes集群中。

    1. 安装NFS客户端。具体操作,请参见安装NFS客户端

    2. 执行以下命令,配置/etc/nfsmount.conf文件。

      cat <<EOF >> /etc/nfsmount.conf
      [ Server "file-system-id.region.nas.aliyuncs.com" ]
      vers=3
      Proto=tcp
      Lock=False
      resvport=False
      rsize=1048576
      wsize=1048576
      hard=True
      timeo=600
      retrans=2
      EOF

      其中,file-system-id.region.nas.aliyuncs.com为NAS文件系统挂载点地址,请根据实际值替换。您可以在NAS控制台,单击目标文件系统,然后单击挂载使用,将鼠标放置在挂载地址列的挂载点图标上获取挂载点地址。

      说明
      • 仅推荐使用NFS v3协议挂载NAS文件系统。

      • 以上配置参数均为必填参数,除file-system-id.region.nas.aliyuncs.com参数和vers协议版本可修改外,其他参数无需修改。

    3. 验证文件系统与Kubernetes节点网络连通。

      1. 执行以下命令,尝试挂载NFS协议文件系统。

        mount -t nfs file-system-id.region.nas.aliyuncs.com:/ /mnt 

        其中,file-system-id.region.nas.aliyuncs.com为NAS文件系统挂载点地址,请根据实际值替换。

        如果挂载失败,请先确认NAS挂载点域名是否能够ping通,并且是否连通2049端口。如果以上问题不存在,请使用检测工具进行排查。具体操作,请参见Linux挂载NFS文件系统挂载失败排查

      2. 执行以下命令,查看挂载信息。

        mount | grep nfs | grep mnt 

        如果挂载参数包含vers=3、hard、noresvport、nolock等信息,则表示网络连通。参数信息

        确认网络连通后,请执行umount /mnt删除挂载的NFS协议文件系统。

  3. 启动应用Pod通过Volume挂载使用NFS文件系统。

    本文以启动一个nginx pod为例介绍通过Volume挂载使用NFS文件系统。

    1. 配置nginx.yaml

      cat  << EOF > ./nginx.yaml
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: nginx-deployment
      spec:
        selector:
          matchLabels:
            app: nginx
        replicas: 1
        template:
          metadata:
            labels:
              app: nginx
          spec:
            containers:
            - name: nginx
              image: nginx:1.14.2
              ports:
              - containerPort: 80
              volumeMounts:
              - mountPath: /data
                name: test-nfs
            volumes:
            - name: test-nfs
              nfs:
                server: file-system-id.region.nas.aliyuncs.com    # 阿里云NAS文件系统挂载点地址,请根据实际值替换。例如,7bexxxxxx-xxxx.ap-southeast-1.nas.aliyuncs.com。
                path: /    # NAS文件系统目录路径。该目录必须为已经存在的目录或根目录。通用型NAS的根目录为“/”,极速型NAS的根目录为“/share”。
      EOF
    2. 启用nginx pod使用NFS文件系统。

      kubectl apply -f ./nginx.yaml
  4. 检查Pod是否正常启动及挂载NFS文件系统成功。

    1. 执行命令,查看Pod运行状态。

      • 执行命令

        kubectl get pods | grep nginx 
      • 返回信息

        如果Pod运行状态为Running,则表示正常启动。Runing

    2. 执行以下命令,查看挂载结果。

      • 执行命令

        kubectl exec nginx-deployment-85c45c8b76-h9vcr -ti stat /data 
      • 返回信息

        如果返回如下类似信息,则表示挂载文件系统成功。挂载成功

方法二:通过CSI插件挂载

重要

CSI插件为非阿里云CSI组件,阿里云不会对该插件的版本、安全性、准确性进行任何保证,在使用过程中阿里云不对由此引发的任何损害承担责任。请您根据实际情况进行选择合适的方式挂载访问NAS。如使用过程中遇到问题,您可以前往GitHub上提交issue进行交流。

  1. 部署CSI插件。

    1. 创建用于部署CSI-ALL的YAML文件。

      将以下内容保存为csi-all.yaml,其中ACCESS_KEY_ID和ACCESS_KEY_SECRET的value请替换为自身阿里云账号(主账号)的AccessKey ID和AccessKey Secret,获取方式请参见创建AccessKey

      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: csi-admin
        namespace: kube-system
      ---
      kind: ClusterRole
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: alicloud-csi-plugin
      rules:
        - apiGroups: [""]
          resources: ["secrets"]
          verbs: ["get", "create", "list"]
        - apiGroups: [""]
          resources: ["persistentvolumes"]
          verbs: ["get", "list", "watch", "update", "create", "delete", "patch"]
        - apiGroups: [""]
          resources: ["persistentvolumeclaims"]
          verbs: ["get", "list", "watch", "update"]
        - apiGroups: [""]
          resources: ["persistentvolumeclaims/status"]
          verbs: ["get", "list", "watch", "update", "patch"]
        - apiGroups: ["storage.k8s.io"]
          resources: ["storageclasses"]
          verbs: ["get", "list", "watch"]
        - apiGroups: ["storage.k8s.io"]
          resources: ["csinodes"]
          verbs: ["get", "list", "watch"]
        - apiGroups: [""]
          resources: ["events"]
          verbs: ["get", "list", "watch", "create", "update", "patch"]
        - apiGroups: [""]
          resources: ["endpoints"]
          verbs: ["get", "watch", "list", "delete", "update", "create"]
        - apiGroups: [""]
          resources: ["configmaps"]
          verbs: ["get", "watch", "list", "delete", "update", "create"]
        - apiGroups: [""]
          resources: ["nodes"]
          verbs: ["get", "list", "watch", "update"]
        - apiGroups: ["csi.storage.k8s.io"]
          resources: ["csinodeinfos"]
          verbs: ["get", "list", "watch"]
        - apiGroups: ["storage.k8s.io"]
          resources: ["volumeattachments"]
          verbs: ["get", "list", "watch", "update", "patch"]
        - apiGroups: ["snapshot.storage.k8s.io"]
          resources: ["volumesnapshotclasses"]
          verbs: ["get", "list", "watch", "create"]
        - apiGroups: ["snapshot.storage.k8s.io"]
          resources: ["volumesnapshotcontents"]
          verbs: ["create", "get", "list", "watch", "update", "delete"]
        - apiGroups: ["snapshot.storage.k8s.io"]
          resources: ["volumesnapshots"]
          verbs: ["get", "list", "watch", "update", "create"]
        - apiGroups: ["apiextensions.k8s.io"]
          resources: ["customresourcedefinitions"]
          verbs: ["create", "list", "watch", "delete", "get", "update", "patch"]
        - apiGroups: ["coordination.k8s.io"]
          resources: ["leases"]
          verbs: ["get", "create", "list", "watch", "delete", "update"]
        - apiGroups: ["snapshot.storage.k8s.io"]
          resources: ["volumesnapshotcontents/status"]
          verbs: ["update"]
        - apiGroups: ["storage.k8s.io"]
          resources: ["volumeattachments/status"]
          verbs: ["patch"]
        - apiGroups: ["snapshot.storage.k8s.io"]
          resources: ["volumesnapshots/status"]
          verbs: ["update"]
        - apiGroups: ["storage.k8s.io"]
          resources: ["storageclasses"]
          verbs: ["get", "list", "watch"]
        - apiGroups: [""]
          resources: ["namespaces"]
          verbs: ["get", "list"]
        - apiGroups: [""]
          resources: ["pods","pods/exec"]
          verbs: ["create", "delete", "get", "post", "list", "watch", "patch", "udpate"]
        - apiGroups: ["storage.alibabacloud.com"]
          resources: ["rules"]
          verbs: ["get"]
        - apiGroups: ["storage.alibabacloud.com"]
          resources: ["containernetworkfilesystems"]
          verbs: ["get","list", "watch"]
      ---
      kind: ClusterRoleBinding
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: alicloud-csi-plugin
      subjects:
        - kind: ServiceAccount
          name: csi-admin
          namespace: kube-system
      roleRef:
        kind: ClusterRole
        name: alicloud-csi-plugin
        apiGroup: rbac.authorization.k8s.io
      ---
      apiVersion: storage.k8s.io/v1
      kind: CSIDriver
      metadata:
        name: diskplugin.csi.alibabacloud.com
      spec:
        attachRequired: true
        podInfoOnMount: true
      ---
      apiVersion: storage.k8s.io/v1
      kind: CSIDriver
      metadata:
        name: nasplugin.csi.alibabacloud.com
      spec:
        attachRequired: false
        podInfoOnMount: true
      ---
      apiVersion: storage.k8s.io/v1
      kind: CSIDriver
      metadata:
        name: ossplugin.csi.alibabacloud.com
      spec:
        attachRequired: false
        podInfoOnMount: true
      ---
      kind: DaemonSet
      apiVersion: apps/v1
      metadata:
        name: csi-plugin
        namespace: kube-system
      spec:
        selector:
          matchLabels:
            app: csi-plugin
        template:
          metadata:
            labels:
              app: csi-plugin
          spec:
            tolerations:
              - operator: Exists
            affinity:
              nodeAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                  - matchExpressions:
                    - key: type
                      operator: NotIn
                      values:
                      - virtual-kubelet
            nodeSelector:
              kubernetes.io/os: linux
            serviceAccount: csi-admin
            priorityClassName: system-node-critical
            hostNetwork: true
            hostPID: true
            dnsPolicy: ClusterFirst
            containers:
              - name: disk-driver-registrar
                image: registry.cn-beijing.aliyuncs.com/acs/csi-node-driver-registrar:v2.3.1-038aeb6-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--v=5"
                  - "--csi-address=/var/lib/kubelet/csi-plugins/diskplugin.csi.alibabacloud.com/csi.sock"
                  - "--kubelet-registration-path=/var/lib/kubelet/csi-plugins/diskplugin.csi.alibabacloud.com/csi.sock"
                volumeMounts:
                  - name: kubelet-dir
                    mountPath: /var/lib/kubelet
                  - name: registration-dir
                    mountPath: /registration
              - name: nas-driver-registrar
                image: registry.cn-beijing.aliyuncs.com/acs/csi-node-driver-registrar:v2.3.1-038aeb6-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--v=5"
                  - "--csi-address=/var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/csi.sock"
                  - "--kubelet-registration-path=/var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/csi.sock"
                volumeMounts:
                  - name: kubelet-dir
                    mountPath: /var/lib/kubelet/
                  - name: registration-dir
                    mountPath: /registration
              - name: oss-driver-registrar
                image: registry.cn-beijing.aliyuncs.com/acs/csi-node-driver-registrar:v2.3.1-038aeb6-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--v=5"
                  - "--csi-address=/var/lib/kubelet/csi-plugins/ossplugin.csi.alibabacloud.com/csi.sock"
                  - "--kubelet-registration-path=/var/lib/kubelet/csi-plugins/ossplugin.csi.alibabacloud.com/csi.sock"
                volumeMounts:
                  - name: kubelet-dir
                    mountPath: /var/lib/kubelet/
                  - name: registration-dir
                    mountPath: /registration
              - name: csi-plugin
                securityContext:
                  privileged: true
                  allowPrivilegeEscalation: true
                image: registry.cn-beijing.aliyuncs.com/acs/csi-plugin:v1.24.5-39a3970-aliyun
                args:
                  - "--endpoint=$(CSI_ENDPOINT)"
                  - "--v=2"
                  - "--driver=oss,nas,disk"
                env:
                  - name: KUBE_NODE_NAME
                    valueFrom:
                      fieldRef:
                        apiVersion: v1
                        fieldPath: spec.nodeName
                  - name: CSI_ENDPOINT
                    value: unix://var/lib/kubelet/csi-plugins/driverplugin.csi.alibabacloud.com-replace/csi.sock
                  - name: MAX_VOLUMES_PERNODE
                    value: "15"
                  - name: SERVICE_TYPE
                    value: "plugin"
                  - name: ACCESS_KEY_ID
                    value: "xxx"
                  - name: ACCESS_KEY_SECRET
                    value: "xxx"
                  - name: KUBELET_ROOT_DIR
                    value: "/var/lib/kubelet"
                resources:
                  requests:
                    cpu: 100m
                    memory: 128Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                livenessProbe:
                  httpGet:
                    path: /healthz
                    port: healthz
                    scheme: HTTP
                  initialDelaySeconds: 10
                  periodSeconds: 30
                  timeoutSeconds: 5
                  failureThreshold: 5
                readinessProbe:
                  httpGet:
                    path: /healthz
                    port: healthz
                  initialDelaySeconds: 10
                  periodSeconds: 30
                  timeoutSeconds: 5
                  failureThreshold: 5
                ports:
                  - name: healthz
                    containerPort: 11260
                volumeMounts:
                  - name: kubelet-dir
                    mountPath: /var/lib/kubelet/
                    mountPropagation: "Bidirectional"
                  - name: etc
                    mountPath: /host/etc
                  - name: host-log
                    mountPath: /var/log/
                  - name: ossconnectordir
                    mountPath: /host/usr/
                  - name: container-dir
                    mountPath: /var/lib/container
                    mountPropagation: "Bidirectional"
                  - name: host-dev
                    mountPath: /dev
                    mountPropagation: "HostToContainer"
                  - mountPath: /host/var/run/
                    name: fuse-metrics-dir
            volumes:
              - name: fuse-metrics-dir
                hostPath:
                  path: /var/run/
                  type: DirectoryOrCreate
              - name: registration-dir
                hostPath:
                  path: /var/lib/kubelet/plugins_registry
                  type: DirectoryOrCreate
              - name: container-dir
                hostPath:
                  path: /var/lib/container
                  type: DirectoryOrCreate
              - name: kubelet-dir
                hostPath:
                  path: /var/lib/kubelet
                  type: Directory
              - name: host-dev
                hostPath:
                  path: /dev
              - name: host-log
                hostPath:
                  path: /var/log/
              - name: etc
                hostPath:
                  path: /etc
              - name: ossconnectordir
                hostPath:
                  path: /usr/
        updateStrategy:
          rollingUpdate:
            maxUnavailable: 30%
          type: RollingUpdate
      
      ---
      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
        name: alicloud-disk-topology-alltype
      parameters:
        type: cloud_essd,cloud_ssd,cloud_efficiency
      provisioner: diskplugin.csi.alibabacloud.com
      reclaimPolicy: Delete
      allowVolumeExpansion: true
      volumeBindingMode: WaitForFirstConsumer
      ---
      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
         name: alicloud-disk-available
      provisioner: diskplugin.csi.alibabacloud.com
      parameters:
          type: available
      reclaimPolicy: Delete
      allowVolumeExpansion: true
      ---
      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
         name: alicloud-disk-essd
      provisioner: diskplugin.csi.alibabacloud.com
      parameters:
          type: cloud_essd
      reclaimPolicy: Delete
      allowVolumeExpansion: true
      ---
      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
         name: alicloud-disk-ssd
      provisioner: diskplugin.csi.alibabacloud.com
      parameters:
          type: cloud_ssd
      reclaimPolicy: Delete
      allowVolumeExpansion: true
      ---
      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
         name: alicloud-disk-efficiency
      provisioner: diskplugin.csi.alibabacloud.com
      parameters:
          type: cloud_efficiency
      reclaimPolicy: Delete
      allowVolumeExpansion: true
      ---
      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
         name: alicloud-disk-topology
      provisioner: diskplugin.csi.alibabacloud.com
      parameters:
          type: available
      reclaimPolicy: Delete
      volumeBindingMode: WaitForFirstConsumer
      allowVolumeExpansion: true
      ---
      kind: Deployment
      apiVersion: apps/v1
      metadata:
        name: csi-provisioner
        namespace: kube-system
      spec:
        selector:
          matchLabels:
            app: csi-provisioner
        strategy:
          rollingUpdate:
            maxSurge: 0
            maxUnavailable: 1
          type: RollingUpdate
        replicas: 2
        template:
          metadata:
            labels:
              app: csi-provisioner
          spec:
            affinity:
              nodeAffinity:
                preferredDuringSchedulingIgnoredDuringExecution:
                - weight: 1
                  preference:
                    matchExpressions:
                    - key: node-role.kubernetes.io/master
                      operator: Exists
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                  - matchExpressions:
                    - key: type
                      operator: NotIn
                      values:
                      - virtual-kubelet
              podAntiAffinity:
                preferredDuringSchedulingIgnoredDuringExecution:
                - weight: 100
                  podAffinityTerm:
                    labelSelector:
                      matchExpressions:
                      - key: app
                        operator: In
                        values:
                        - csi-provisioner
                    topologyKey: kubernetes.io/hostname
            tolerations:
            - effect: NoSchedule
              operator: Exists
              key: node-role.kubernetes.io/master
            - effect: NoSchedule
              operator: Exists
              key: node.cloudprovider.kubernetes.io/uninitialized
            serviceAccount: csi-admin
            hostPID: true
            priorityClassName: system-node-critical
            containers:
              - name: external-disk-provisioner
                image: registry.cn-beijing.aliyuncs.com/acs/csi-provisioner:v3.0.0-080f01e64-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--csi-address=$(ADDRESS)"
                  - "--feature-gates=Topology=True"
                  - "--volume-name-prefix=disk"
                  - "--strict-topology=true"
                  - "--timeout=150s"
                  - "--leader-election=true"
                  - "--retry-interval-start=500ms"
                  - "--extra-create-metadata=true"
                  - "--default-fstype=ext4"
                  - "--v=5"
                env:
                  - name: ADDRESS
                    value: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com/csi.sock
                volumeMounts:
                  - name: disk-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com
              - name: external-disk-attacher
                image: registry.cn-beijing.aliyuncs.com/acs/csi-attacher:v3.3-72dd428b-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--v=5"
                  - "--csi-address=$(ADDRESS)"
                  - "--leader-election=true"
                env:
                  - name: ADDRESS
                    value: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com/csi.sock
                volumeMounts:
                  - name: disk-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com
              - name: external-disk-resizer
                image: registry.cn-beijing.aliyuncs.com/acs/csi-resizer:v1.3-ca84e84-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--v=5"
                  - "--csi-address=$(ADDRESS)"
                  - "--leader-election"
                env:
                  - name: ADDRESS
                    value: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com/csi.sock
                volumeMounts:
                  - name: disk-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com
              - name: external-nas-provisioner
                image: registry.cn-beijing.aliyuncs.com/acs/csi-provisioner:v3.0.0-080f01e64-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--csi-address=$(ADDRESS)"
                  - "--volume-name-prefix=nas"
                  - "--timeout=150s"
                  - "--leader-election=true"
                  - "--retry-interval-start=500ms"
                  - "--default-fstype=nfs"
                  - "--v=5"
                env:
                  - name: ADDRESS
                    value: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com/csi.sock
                volumeMounts:
                  - name: nas-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com
              - name: external-nas-resizer
                image: registry.cn-beijing.aliyuncs.com/acs/csi-resizer:v1.3-ca84e84-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--v=5"
                  - "--csi-address=$(ADDRESS)"
                  - "--leader-election"
                env:
                  - name: ADDRESS
                    value: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com/csi.sock
                volumeMounts:
                  - name: nas-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com
              - name: external-oss-provisioner
                args:
                  - --csi-address=$(ADDRESS)
                  - --volume-name-prefix=oss
                  - --timeout=150s
                  - --leader-election=true
                  - --retry-interval-start=500ms
                  - --default-fstype=ossfs
                  - --v=5
                env:
                - name: ADDRESS
                  value: /var/lib/kubelet/csi-provisioner/ossplugin.csi.alibabacloud.com/csi.sock
                image: registry.cn-beijing.aliyuncs.com/acs/csi-provisioner:v3.0.0-080f01e64-aliyun
                resources:
                  limits:
                    cpu: 500m
                    memory: 1Gi
                  requests:
                    cpu: 10m
                    memory: 16Mi
                volumeMounts:
                - mountPath: /var/lib/kubelet/csi-provisioner/ossplugin.csi.alibabacloud.com
                  name: oss-provisioner-dir
              - name: external-csi-snapshotter
                image: registry.cn-beijing.aliyuncs.com/acs/csi-snapshotter:v4.0.0-a230d5b3-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--v=5"
                  - "--csi-address=$(ADDRESS)"
                  - "--leader-election=true"
                  - "--extra-create-metadata=true"
                env:
                  - name: ADDRESS
                    value: /csi/csi.sock
                volumeMounts:
                  - name: disk-provisioner-dir
                    mountPath: /csi
              - name: external-snapshot-controller
                image: registry.cn-beijing.aliyuncs.com/acs/snapshot-controller:v4.0.0-a230d5b3-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--v=5"
                  - "--leader-election=true"
              - name: csi-provisioner
                securityContext:
                  privileged: true
                image: registry.cn-beijing.aliyuncs.com/acs/csi-plugin:v1.24.5-39a3970-aliyun
                args:
                  - "--endpoint=$(CSI_ENDPOINT)"
                  - "--v=2"
                  - "--driver=nas,disk,oss"
                env:
                  - name: CSI_ENDPOINT
                    value: unix://var/lib/kubelet/csi-provisioner/driverplugin.csi.alibabacloud.com-replace/csi.sock
                  - name: MAX_VOLUMES_PERNODE
                    value: "15"
                  - name: SERVICE_TYPE
                    value: "provisioner"
                  - name: "CLUSTER_ID"
                    value: "abc"
                  - name: "ACCESS_KEY_ID"
                    value: "xxx"
                  - name: "ACCESS_KEY_SECRET"
                    value: "xxx"
      
                livenessProbe:
                  httpGet:
                    path: /healthz
                    port: healthz
                    scheme: HTTP
                  initialDelaySeconds: 10
                  periodSeconds: 30
                  timeoutSeconds: 5
                  failureThreshold: 5
                readinessProbe:
                  httpGet:
                    path: /healthz
                    port: healthz
                  initialDelaySeconds: 5
                  periodSeconds: 20
                ports:
                  - name: healthz
                    containerPort: 11270
                volumeMounts:
                  - name: host-log
                    mountPath: /var/log/
                  - name: disk-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com
                  - name: nas-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com
                  - name: oss-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/ossplugin.csi.alibabacloud.com
                  - mountPath: /var/addon
                    name: addon-token
                    readOnly: true
                  - mountPath: /mnt
                    mountPropagation: Bidirectional
                    name: host-dev
                  - mountPath: /host/etc
                    name: etc
                resources:
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                  requests:
                    cpu: 100m
                    memory: 128Mi
            volumes:
              - name: disk-provisioner-dir
                emptyDir: {}
              - name: nas-provisioner-dir
                emptyDir: {}
              - name: oss-provisioner-dir
                emptyDir: {}
              - name: host-log
                hostPath:
                  path: /var/log/
              - name: etc
                hostPath:
                  path: /etc
                  type: ""
              - name: host-dev
                hostPath:
                  path: /mnt
                  type: ""
              - name: addon-token
                secret:
                  defaultMode: 420
                  optional: true
                  items:
                  - key: addon.token.config
                    path: token-config
                  secretName: addon.csi.token
    2. 部署组件。

      kubectl apply -f csi-all.yaml
    3. 查看部署结果。

      kubectl -n kube-system get pods

      预期返回如下:

      NAME                                    READY   STATUS    RESTARTS   AGE
      csi-plugin-5h6gq                       4/4     Running   0          30m
      csi-plugin-gjjp7                       4/4     Running   0          31m
      csi-provisioner-cfc45d84c-lzpvm        9/9     Running   0          31m
      csi-provisioner-cfc45d84c-zczfn        9/9     Running   0          31m
      ......
  2. 挂载文件系统。