ossfs 1.0 supports dynamic provisioning, which automatically creates a persistent volume (PV) and mounts an OSS Bucket based on a StorageClass and a PersistentVolumeClaim (PVC). It eliminates manual PV configuration, greatly simplifies storage management, and is ideal for multi-tenant environments or scenarios requiring frequent, on-demand storage provisioning.
Requirements
Your cluster and CSI components (csi-plugin and csi-provisioner) must meet the following requirements:
For authentication using RAM Roles for Service Accounts (RRSA): Your cluster must be version 1.26 or later, and the CSI version must be v1.30.4 or later.
If you used the RRSA feature in a version earlier than v1.30.4, refer to [Product Change] CSI ossfs version upgrade and mount process optimization to add the required RAM Role authorization configuration.
For authentication using an AccessKey: To ensure mount stability, we recommend CSI version v1.18.8.45 or later.
To upgrade your cluster, see Manually upgrade a cluster. To upgrade the CSI components, see Upgrade CSI components.
Starting with CSI v1.30.4-*, mounting OSS static volumes depends on the csi-provisioner component.
Step 1: Choose an authentication method
To ensure that your cluster can access OSS Bucket resources securely and compliantly, you must first configure an authentication method.
RRSA authentication: Dynamically grants Pods temporary, automatically rotating RAM roles to enable fine-grained, application-level permission isolation. This method is more secure.
AccessKey authentication: Stores static, long-term keys in a Secret. This method is simpler to configure but less secure.
For clusters running version 1.26 or later, we recommend RRSA authentication to prevent workload restarts and ossfs remounts caused by AccessKey rotation.
This example assumes that the cluster and the OSS Bucket are in the same Alibaba Cloud account. To mount an OSS Bucket that belongs to another account, we recommend using RRSA authentication.
RRSA
1. Enable RRSA for your cluster
On the ACK Clusters page, click the name of your cluster. In the left navigation pane, click Cluster Information.
On the Basic Information tab, in the Security and Auditing section, click Enabled to the right of RRSA OIDC. Follow the on-screen instructions to enable RRSA during off-peak hours.
When the cluster status changes from Updating to Running, RRSA is enabled.
ImportantAfter you enable RRSA, the maximum validity period of newly created ServiceAccount tokens in the cluster is limited to 12 hours.
2. Create and authorize a RAM role
Create a RAM role that your Pods can assume for RRSA-authenticated access to the OSS storage volume.
AccessKey
Create a RAM user with permissions to access the target OSS Bucket, and then obtain an AccessKey for that user.
Create a RAM user. You can skip this step if you already have a RAM user.
Go to the Create User page on the RAM Console and follow the on-screen instructions to create a RAM user. You must specify parameters such as the logon name and password.
Create an access policy.
This example follows the principle of least privilege. Create a custom policy to grant access to the target OSS Bucket. You can grant read-only or read/write permissions.
Go to the Create Policy page on the RAM Console, switch to the JSON Editor, and configure the policy script.
Read-only policy
Replace
<myBucketName>with the actual bucket name.{ "Statement": [ { "Action": [ "oss:Get*", "oss:List*" ], "Effect": "Allow", "Resource": [ "acs:oss:*:*:<myBucketName>", "acs:oss:*:*:<myBucketName>/*" ] } ], "Version": "1" }OSS read/write policy
Replace
<myBucketName>with the actual bucket name.{ "Statement": [ { "Action": "oss:*", "Effect": "Allow", "Resource": [ "acs:oss:*:*:<myBucketName>", "acs:oss:*:*:<myBucketName>/*" ] } ], "Version": "1" }If you want to create a PV in the console, you also need the
oss:ListBucketspermission.{ "Effect": "Allow", "Action": "oss:ListBuckets", "Resource": "*" }(Optional) If you use a specific CMK ID managed by KMS to encrypt OSS objects, you must also grant KMS permissions to the RAM user. For more information, see Use a specific CMK ID managed by KMS to encrypt data.
Attach the access policy to the RAM user.
Go to the Users page on the RAM Console, find the user you created, and in the Actions column, click Add Permissions.
In the Policies section, search for and select the access policy you created.
The AccessKey you create will be stored as a Secret for the PV.
Go to the Users page on the RAM Console, click the name of your RAM user, and then in the AccessKey section, click Create AccessKey.
Follow the on-screen instructions to create an AccessKey. Copy and securely store the AccessKey ID and AccessKey Secret.
Step 2: Create a StorageClass
Create a StorageClass to define a template for provisioning persistent volumes.
RRSA method
-
Create a file named
sc-oss.yaml.apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: sc-oss parameters: # Replace with the name of your OSS bucket. bucket: bucket # The mount path relative to the bucket's root directory. You can specify a subdirectory. path: / # The endpoint of the region where the bucket is located. url: "http://oss-cn-hangzhou-internal.aliyuncs.com" # Use the RRSA method for authentication. authType: rrsa # The RAM role that you created or modified. roleName: demo-role-for-rrsa # Custom parameters. otherOpts: "-o umask=022 -o max_stat_cache_size=100000 -o allow_other" # The volume access mode. volumeAs: sharepath # Fixed value for the Alibaba Cloud OSS CSI plug-in. provisioner: ossplugin.csi.alibabacloud.com # The reclaim policy for the dynamically provisioned persistent volume. reclaimPolicy: Retain # The volume binding mode. volumeBindingMode: ImmediateParameter
Description
bucketThe OSS bucket to mount.
pathThe version of the CSI component must be v1.14.8.32-c77e277b-aliyun or later.
The mount path relative to the root directory of the bucket. The default value is
/, which mounts the entire bucket.If the ossfs version is earlier than 1.91, the specified
pathmust exist in the OSS bucket. For more information, see New features in ossfs 1.91 and later.urlThe endpoint of the OSS Bucket to mount.
Use an internal endpoint if the mount node and the bucket are in the same region, or if a VPC connection is established.
Use a public endpoint if the mount node and the bucket are in different regions.
The following are common formats for different access ports:
Internal network format:
http://oss-{{regionName}}-internal.aliyuncs.comorhttps://oss-{{regionName}}-internal.aliyuncs.com.The internal endpoint format
vpc100-oss-{{regionName}}.aliyuncs.comis deprecated. Switch to the new format promptly.Public network format:
http://oss-{{regionName}}.aliyuncs.comorhttps://oss-{{regionName}}.aliyuncs.com.
authTypeSet this parameter to
rrsato use RRSA for authentication.roleNameThe RAM role that you created or modified.
To configure different permissions for different StorageClasses, you can create multiple RAM roles and specify different
roleNamevalues in each StorageClass.otherOptsCustom parameters for the OSS persistent volume, in the format of
-o *** -o ***. For example:-o umask=022 -o max_stat_cache_size=100000 -o allow_other.provisionerThe driver type. This value is fixed at
ossplugin.csi.alibabacloud.comwhen using the Alibaba Cloud OSS CSI plug-in.reclaimPolicyThe reclaim policy for dynamically provisioned persistent volumes. OSS persistent volumes support only
Retain, which means that when you delete a PVC, the corresponding persistent volume and the data in the OSS bucket are not deleted.volumeBindingModeThe volume binding mode.
OSS persistent volumes do not require availability zone-based node affinity. You can use the default value
Immediate.volumeAsThe volume access mode. The default value is
sharepath. Valid values:-
sharepath: All persistent volumes share the same mount path. Data is stored at<bucket>:<path>/. -
subpath: Creates a dedicated subdirectory for each persistent volume to isolate data. The data is stored at<bucket>:<path>/<pv-name>/.This parameter takes effect only when the CSI component version is 1.31.3 or later.
sigVersionThe signature version for requests sent to the OSS server.
"v1"(Default): Uses OSS Signature Version 1."v4"(Recommended): Uses OSS Signature Version 4.
-
Create the StorageClass.
kubectl apply -f sc-oss.yaml
AccessKey method
kubectl
1. Create a StorageClass
-
Create a Secret in the same namespace as your application.
Replace
akIdandakSecretwith your AccessKey ID and AccessKey Secret.kubectl create secret generic oss-secret --from-literal='akId=<yourAccessKey ID>' --from-literal='akSecret=<yourAccessKey Secret>' -
Create the StorageClass.
-
Create a file named
sc-oss.yaml.apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: sc-oss parameters: # Replace with the name of your OSS bucket. bucket: bucket # The mount path relative to the bucket's root directory. You can specify a subdirectory. path: / # The endpoint of the region where the bucket is located. url: "http://oss-cn-hangzhou-internal.aliyuncs.com" # The name of the Secret that stores the AccessKey information. csi.storage.k8s.io/node-publish-secret-name: oss-secret # The namespace where the Secret that stores the AccessKey information is located. csi.storage.k8s.io/node-publish-secret-namespace: default # Custom parameters. otherOpts: "-o umask=022 -o max_stat_cache_size=100000 -o allow_other" # Fixed value for the Alibaba Cloud OSS CSI plug-in. provisioner: ossplugin.csi.alibabacloud.com # The reclaim policy for the dynamically provisioned persistent volume. reclaimPolicy: Retain # The volume binding mode. volumeBindingMode: ImmediateParameter
Description
nameThe name of the StorageClass.
bucketThe OSS bucket to mount.
pathThe version of the CSI component must be v1.14.8.32-c77e277b-aliyun or later.
The mount path relative to the root directory of the bucket. The default value is
/, which mounts the entire bucket.If the ossfs version is earlier than 1.91, the specified
pathmust exist in the OSS bucket. For more information, see New features in ossfs 1.91 and later.urlThe endpoint of the OSS Bucket to mount.
Use an internal endpoint if the mount node and the bucket are in the same region, or if a VPC connection is established.
Use a public endpoint if the mount node and the bucket are in different regions.
The following are common formats for different access ports:
Internal network format:
http://oss-{{regionName}}-internal.aliyuncs.comorhttps://oss-{{regionName}}-internal.aliyuncs.com.The internal endpoint format
vpc100-oss-{{regionName}}.aliyuncs.comis deprecated. Switch to the new format promptly.Public network format:
http://oss-{{regionName}}.aliyuncs.comorhttps://oss-{{regionName}}.aliyuncs.com.
csi.storage.k8s.io/node-publish-secret-nameThe name of the Secret that stores the AccessKey information.
csi.storage.k8s.io/node-publish-secret-namespaceThe namespace where the Secret that stores the AccessKey information is located.
otherOptsCustom parameters for the OSS persistent volume, in the format of
-o *** -o ***. For example:-o umask=022 -o max_stat_cache_size=100000 -o allow_other.provisionerThe driver type. This value is fixed at
ossplugin.csi.alibabacloud.comwhen using the Alibaba Cloud OSS CSI plug-in.reclaimPolicyThe reclaim policy for dynamically provisioned persistent volumes. OSS persistent volumes support only
Retain, which means that when you delete a PVC, the corresponding persistent volume and the data in the OSS bucket are not deleted.volumeBindingModeThe volume binding mode.
OSS persistent volumes do not require availability zone-based node affinity. You can use the default value
Immediate.volumeAsThe volume access mode. The default value is
sharepath. Valid values:-
sharepath: All persistent volumes share the same mount path. Data is stored at<bucket>:<path>/. -
subpath: Creates a dedicated subdirectory for each persistent volume to isolate data. The data is stored at<bucket>:<path>/<pv-name>/.This parameter takes effect only when the CSI component version is 1.31.3 or later.
sigVersionThe signature version for requests sent to the OSS server.
"v1"(Default): Uses OSS Signature Version 1."v4"(Recommended): Uses OSS Signature Version 4.
-
Create the StorageClass.
kubectl apply -f sc-oss.yaml
-
Console
-
Store the AccessKey from Step 1 as a Secret for the persistent volume.
On the ACK Clusters page, click the name of your cluster. In the left navigation pane, click .
Click Create from YAML and follow the on-screen instructions to create the Secret.
apiVersion: v1 kind: Secret metadata: name: oss-secret # Must match the namespace of the application. namespace: default stringData: # Replace with your AccessKey ID. akId: <your AccessKey ID> # Replace with your AccessKey secret. akSecret: <your AccessKey Secret>
On the Clusters page, click the name of your cluster. In the left navigation pane, click .
-
On the StorageClasses page, click Create, set PV Type to OSS, and follow the on-screen instructions to configure the StorageClass.
Parameter
Description
Access certificate
The Secret used to access OSS, which contains the AccessKey ID and AccessKey Secret.
Bucket ID:
The OSS bucket to use.
The console displays only the buckets that the configured AccessKey can access.
OSS path
The version of the CSI component must be v1.14.8.32-c77e277b-aliyun or later.
The mount path relative to the root directory of the bucket. The default value is
/, which mounts the entire bucket.If the ossfs version is earlier than 1.91, the specified
pathmust exist in the OSS bucket. For more information, see New features in ossfs 1.91 and later.Volume Mode
The volume access mode. The default is Shared Directory. Valid values:
The Subdirectory mode takes effect only when the CSI component version is 1.31.3 or later. Otherwise, the Shared Directory mode is used.
-
Shared Directory (
sharepath): All persistent volumes share the same mount path. Data is stored at<bucket>:<path>/. -
Subdirectory (
subpath): Creates a dedicated subdirectory for each persistent volume to isolate data. Data is stored at<bucket>:<path>/<pv-name>/.
Endpoint
The endpoint of the OSS Bucket to mount.
Use an internal endpoint if the mount node and the bucket are in the same region, or if a VPC connection is established.
Use a public endpoint if the mount node and the bucket are in different regions.
The following are common formats for different access ports:
Internal network format:
http://oss-{{regionName}}-internal.aliyuncs.comorhttps://oss-{{regionName}}-internal.aliyuncs.com.The internal endpoint format
vpc100-oss-{{regionName}}.aliyuncs.comis deprecated. Switch to the new format promptly.Public network format:
http://oss-{{regionName}}.aliyuncs.comorhttps://oss-{{regionName}}.aliyuncs.com.
By default, the console uses HTTP to access OSS over an internal network. To use HTTPS, you must create the PV by using kubectl.
Reclaim policy
The reclaim policy for dynamically provisioned persistent volumes. OSS persistent volumes support only
Retain, which means that when you delete a PVC, the corresponding persistent volume and the data in the OSS bucket are not deleted.Optional Parameters:
Custom parameters for the OSS persistent volume, in the format of
-o *** -o ***. For example:-o umask=022 -o max_stat_cache_size=100000 -o allow_other. -
Step 3: Create a PVC
Create a PVC to dynamically request storage resources. The CSI plug-in automatically creates a PV based on the StorageClass.
kubectl
-
Create a file named
pvc-oss.yaml:apiVersion: v1 kind: PersistentVolumeClaim metadata: # PVC name name: pvc-oss spec: # Configure the access mode. ReadOnlyMany means that ossfs mounts the OSS bucket in read-only mode. accessModes: - ReadOnlyMany volumeMode: Filesystem resources: requests: # Declare the requested storage capacity. This value does not limit the actual capacity of the OSS volume. storage: 20Gi # Specify the StorageClass to use. storageClassName: sc-ossParameter
Description
accessModesSpecifies the access mode. Supported values are
ReadOnlyManyandReadWriteMany.If set to
ReadOnlyMany, ossfs mounts the OSS bucket in read-only mode.storageThe requested capacity of the volume. This value does not limit the actual capacity of the OSS volume.
storageClassNameThe name of the StorageClass.
-
Create the PVC:
kubectl apply -f pvc-oss.yaml -
Verify that the PVC is created and in the Bound state:
kubectl get pvc pvc-ossExpected output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE pvc-oss Bound oss-251d111d-3b0b-4879-81a0-eb5a19xxxxxx 20Gi ROX sc-oss <unset> 4d20h
Console
On the Clusters page, click the name of your cluster. In the left navigation pane, click .
-
On the Persistent Volume Claims page, click Create. Set PVC Type to OSS and configure the parameters.
Parameter
Description
Allocation Mode
Select Use StorageClass.
Existing Storage Class
Click Select Storage Class and choose the previously created StorageClass.
Capacity
The requested capacity of the volume. This value does not limit the actual capacity of the OSS volume.
Access Mode
Specifies the access mode. Supported values are
ReadOnlyManyandReadWriteMany.If set to
ReadOnlyMany, ossfs mounts the OSS bucket in read-only mode.
Step 4: Create an application and mount a volume
To mount the volume, reference the PVC in your application.
kubectl
Create an
oss-workload.yamlfile.apiVersion: apps/v1 kind: Deployment metadata: name: oss-workload labels: app: nginx spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6 ports: - containerPort: 80 volumeMounts: # The mount path in the container. - name: pvc-oss mountPath: "/data" # Configure a health check. livenessProbe: exec: command: - ls - /data initialDelaySeconds: 30 periodSeconds: 30 volumes: - name: pvc-oss persistentVolumeClaim: # Reference the PVC that you created. claimName: pvc-ossCreate the application.
kubectl create -f oss-workload.yamlVerify the mount.
Confirm that the Pods are running.
kubectl get pod -l app=nginxAccess a Pod and check the mount point.
kubectl exec -it <pod-name> -- ls /dataThe output should list the files in the OSS mount path.
Console
On the ACK Clusters page, click the name of your cluster. In the left navigation pane, click .
On the Deployments page, click Create from Image.
Click Create from Image and follow the on-screen instructions to configure the application parameters.
The key parameters are described in the following table. For other parameters, you can use the default values. For more information, see Create a Deployment.
Configuration page
Parameter
Description
Basic Information
Replicas:
The number of Pod replicas for the Deployment.
Container
Image name
The address of the container image for the application, such as
anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6.Required Resources
The amount of vCPU and memory to allocate to each container.
Volumes
Click Add PVC and then configure the parameters.
Mount Source: Select the PVC that you created earlier.
Container Path: The path within the container where the volume will be mounted, such as
/data.
Labels and Annotations
Pod label
For example, set the key to 'app' and the value to 'nginx'.
Check the application deployment status.
On the Deployments page, click the name of your application. On the Pods tab, verify that the Pods have a Running status.
Step 5: Verify shared and persistent storage
Verify shared storage
To verify shared storage, create a file in one Pod and then view it in another.
Get the names of the application Pods.
kubectl get pod -l app=nginxThis example uses the Pod
oss-workload-66fbb85b67-d****:ReadWriteMany: Create the tmpfile file in the/datapath.kubectl exec oss-workload-66fbb85b67-d**** -- touch /data/tmpfileReadOnlyMany: Upload thetmpfilefile to the corresponding path in the OSS bucket by using the OSS console or by uploading a file with cp.
Check for the file in the mount path of another Pod.
Take a Pod named
oss-workload-66fbb85b67-l****with the mount pathdataas an example.kubectl exec oss-workload-66fbb85b67-l**** -- ls /data | grep tmpfileThe presence of the file in the output confirms that the Pods can share data.
tmpfileIf you do not see the expected output, make sure that your CSI component version is v1.20.7 or later.
Verify persistent storage
To verify persistent storage, delete a Pod and then check that the file remains after the Pod is automatically recreated.
Delete an application Pod. The Deployment controller automatically recreates it.
kubectl delete pod oss-workload-66fbb85b67-d****Check the status of the Pods. Wait for the new Pod to start and become
Running.kubectl get pod -l app=nginxView the files in the
/datapath.This example uses a Pod named
oss-workload-66fbb85b67-z****with the mount pathdata.kubectl exec oss-workload-66fbb85b67-z**** -- ls /data | grep tmpfileThe output shows the
tmpfilefile still exists, which confirms that data persists even after the Pod is recreated.tmpfile
Known impacts
Data integrity risks
Concurrent write consistency risk: To improve write stability, we recommend upgrading the CSI component to v1.28 or later. However, in single-file concurrent write scenarios, the OSS "overwrite upload" feature can still overwrite data. You must ensure data consistency at the application layer.
Data synchronization and accidental deletion risk: When a volume is mounted, deleting or modifying files in the mount path on an application Pod or host node directly synchronizes those changes with the source files in the OSS Bucket. To prevent accidental data deletion, we recommend enabling versioning for your OSS Bucket.
Application stability risks
OOM risk: When you perform a
readdiroperation (such as thelscommand in a shell script) on a large number of files (for example, more than 100,000, depending on the node memory) for the first time, ossfs loads all metadata at once and consumes a large amount of memory. This may cause the process to be OOM Killed and make the mount point unavailable.To mitigate this risk, we recommend mounting a subdirectory of the OSS Bucket or optimizing your directory structure.
Extended mount time: Configuring
securityContext.fsgroupin an application causes kubelet to recursively modify file permissions (chmod/chown) when mounting a volume. If the volume contains a large number of files, the mount time is significantly extended, which can cause severe Pod startup delays.If you must configure this parameter but want to reduce the mount time, see Increased mount time for OSS volumes.
Key invalidation risk (AccessKey authentication): If an AccessKey referenced by a PV becomes invalid or its permissions change, the associated application immediately loses access.
To restore access, you must update the credentials in the Secret and restart the application Pod to force a remount. This causes a service interruption, so perform this operation during a maintenance window. For more information, see Solutions.
Cost risks
Part costs: When uploading files larger than 10 MB, ossfs performs a multipart upload, splitting the file into parts. If an upload is unexpectedly interrupted, for example, due to an application restart, you must manually delete the parts or delete them by using lifecycle rules to avoid storage costs from incomplete parts.
Resource release
-
Delete workloads
-
Action: Delete all workloads, such as Deployments and StatefulSets, that use the PVC. This stops the Pods and unmounts the volume.
-
Example command:
kubectl delete deployment <your-deployment-name>
-
-
Delete the PVC
-
Action: Delete the PVC. The reclaim policy (
reclaimPolicy) for dynamically provisioned OSS volumes supports onlyRetain. When you delete the PVC, its bound PV enters theReleasedstate, and the backend OSS bucket retains the corresponding subdirectory and data. -
Example command:
kubectl delete pvc <your-pvc-name>
-
-
Delete the PV
-
Action: Delete the PV in the
Releasedstate. This operation only removes the PV resource object from Kubernetes and does not delete any data in the backend OSS bucket. -
Example command:
kubectl delete pv <your-pv-name>
-
-
Delete the StorageClass and Secret (optional)
-
Action: Delete the StorageClass and the associated Secret if they are no longer needed. This operation only removes the resource objects from Kubernetes and does not affect the backend OSS bucket.
-
Example commands:
kubectl delete sc <your-storageclass-name>,kubectl delete secret <your-secret-name>
-
Related documents
Manage OSS volumes by using CNFS to improve their performance and QoS. For more information, see Manage the lifecycle of OSS volumes.
Enable server-side encryption to protect sensitive data at rest in OSS. For more information, see Encrypt ossfs 1.0 volumes.
For frequently asked questions about ossfs and OSS, see ossfs 1.0 (default) and ossfs 1.0 volume FAQ.
Enable container storage monitoring and configure alerts to promptly detect volume anomalies or performance bottlenecks.
ossfs 1.0 provides stronger data consistency for random and concurrent writes than ossfs 2.0. However, ossfs 2.0 offers better performance for sequential reads and writes.