Use a static OSS volume
Object Storage Service (OSS) volumes give your application shared, persistent storage for unstructured data such as images, audio, and video files. Data written to an OSS bucket stays available to every pod that mounts the volume and survives pod restarts.
Background
OSS is a secure, cost-effective, high-capacity, and highly reliable cloud storage solution. It is ideal for storing data that is rarely modified after being written, as well as unstructured data such as images, audio, and video files. For more information, see Storage overview.
OSS volume clients
An OSS volume is a type of volume that uses a Filesystem in Userspace (FUSE) client to mount object storage as a local file system (FS). Compared with traditional local storage and block storage, OSS volumes have some POSIX compatibility limitations. Alibaba Cloud Container Compute Service (ACS) supports the following OSS volume clients.
Use cases | Client | Type | Description |
Most scenarios, including read/write operations and user permission configuration. | FUSE | Supports most POSIX operations, including append write, random write, and setting user permissions. | |
Read-only scenarios or workloads that only involve sequential append writes. Examples include AI model training and inference, big data processing, and autonomous driving. | FUSE | Supports full reads and sequential append writes. Ideal for read-heavy scenarios, such as AI model training and inference, big data processing, and autonomous driving. This client can significantly improve data read performance. |
Before you choose a client, consider the following:
Unknown read/write pattern — If you are unsure about the read/write model of your workload, use ossfs 1.0. It offers better POSIX compatibility and ensures greater operational stability for your workloads.
Separable read and write paths — For scenarios where read and write operations can be separated, such as when reads and writes do not occur simultaneously or target different files (for example, saving checkpoints or persistent logging), use separate volumes. For example, mount a read-only path using an ossfs 2.0 volume and a write path using an ossfs 1.0 volume.
Compute power required by ossfs 2.0 — ossfs 2.0 supports only GPU compute power. To use CPU compute power, submit a ticket to request access.
Mount method supported by ossfs 2.0 — ACS supports mounting static ossfs 2.0 volumes only by using kubectl.
POSIX API support
The following table compares the POSIX API support of ossfs 1.0 and ossfs 2.0.
Feature category | Operation | ossfs 1.0 | ossfs 2.0 |
Basic object operations |
| Supported | Supported |
Basic object operations |
| Supported | Supported |
Basic object operations |
| Supported | Supported |
File reads and writes |
| Supported | Supported |
File reads and writes |
| Supports random writes (requires a disk cache) | Supports only sequential writes (no disk cache required) |
File reads and writes |
| Supported (can be truncated to any size) | Only supports truncating an object to zero length |
File meta operations |
| Supported | Supported |
File meta operations |
| Supported | Supported |
File meta operations |
| Supported | Supported |
Directory operations |
| Supported | Supported |
Directory operations |
| Supported | Supported |
Directory operations |
| Supported | Supported |
Permissions and attributes |
| Supported | Supported |
Permissions and attributes |
| Supported | Unsupported |
Permissions and attributes |
| Supported | Unsupported |
Permissions and attributes |
| Supported | Supported |
Extended features |
| Supported | Unsupported |
Extended features |
| Supported | Unsupported |
Extended features |
| Unsupported | Unsupported |
Performance benchmark
ossfs 2.0 delivers significant performance improvements over ossfs 1.0 in sequential read/write operations and high-concurrency small-file reads.
Sequential write performance: In single-threaded, large-object sequential write scenarios, ossfs 2.0 provides nearly 18 times the bandwidth of ossfs 1.0.
Sequential read performance: In single-threaded or multi-threaded (4 threads) large-object sequential read scenarios, ossfs 2.0 provides more than 3 times the bandwidth of ossfs 1.0.
Concurrent small-object read performance: In high-concurrency (128 threads) small-object read scenarios, ossfs 2.0 provides more than 20 times the bandwidth of ossfs 1.0.
If the read/write performance, such as latency or throughput, does not meet your expectations, see Best practices for tuning the performance of OSS volumes.
Prerequisites
The managed-csiprovisioner add-on is installed in the ACS cluster.
Go to the ACS cluster management page in the ACS console. In the left-side navigation pane of the cluster management page, choose Operations > Add-ons. On the Storagetab, check whether managed-csiprovisioner is installed.
Notes
The following notes mainly apply to general read/write scenarios that use the ossfs 1.0 client. The ossfs 2.0 client is read-focused and supports only a subset of POSIX operations.
ACS supports only OSS static volumes, not dynamic volumes.
Hard links are not supported.
You cannot mount buckets whose storage class is Archive, Cold Archive, or Deep Cold Archive.
Random or append writes to a file create a new file locally and then re-upload it to the OSS server. Due to the nature of object storage, note the following:
Rename operations for files and directories are not atomic.
Avoid concurrent writes or operations such as compression and decompression directly in the mount directory. Coordinate concurrent writes on the client side, for example by serializing the writes that target the same object. Otherwise, concurrent writes, compression, and decompression in the mount directory can leave data or metadata inconsistent, and ACS does not protect against such inconsistencies.
By default, the
readdiroperation on ossfs 1.0 volumes sends numerousHeadObjectrequests to retrieve extended information for all objects in a directory, which can degrade overall ossfs performance in directories with many files. If your workload is not sensitive to file attributes such as permissions, you can enable the-o readdir_optimizeparameter to improve performance. For more information, see readdir optimization.
Create an OSS bucket and get details
Create an OSS bucket.
Log on to the OSS console. In the left-side navigation pane, click Buckets.
Click Create Bucket.
In the Create Bucket panel, configure the required parameters and click Create.
The following table describes the key parameters. For more information, see Create buckets.
Parameter
Description
Bucket Name
Enter a custom name. The bucket name must be globally unique in OSS and cannot be changed after creation. Follow the formatting requirements displayed in the UI.
Region
Select Specific Region and choose the same region as your ACS cluster. This allows pods in the cluster to access the bucket over the internal network.
(Optional) To mount a subdirectory from an OSS bucket, create the subdirectory first.
On the Buckets page, click the name of the target bucket.
On the bucket details page, in the left-side navigation pane, choose Object Management > Objects.
Click Create Directory to create directories in the bucket as needed.
Get the endpoint of the OSS bucket.
On the Buckets page, click the name of the target bucket.
On the bucket details page, click the Overview tab. In the Port section, copy the endpoint that matches your deployment:
If the OSS bucket and the ACS cluster are in the same region, copy the internal endpoint.
If the OSS bucket is region-agnostic or is in a different region from the ACS cluster, copy the public endpoint.
Get an AccessKey ID and an AccessKey Secret to access OSS. For more information, see Obtain an AccessKey pair.
If you need to mount an OSS bucket that belongs to another Alibaba Cloud account, you must get the AccessKey pair from that account.
Mount an OSS volume
Select the tab for the client that you chose. Each procedure creates a Persistent Volume (PV) and a Persistent Volume Claim (PVC), and then mounts the volume in an application.
ossfs 1.0 volume
kubectl
Step 1: Create a PV
Save the following YAML content as
oss-pv.yaml.apiVersion: v1 kind: Secret metadata: name: oss-secret namespace: default stringData: akId: <your AccessKey ID> akSecret: <your AccessKey Secret> --- apiVersion: v1 kind: PersistentVolume metadata: name: oss-pv labels: alicloud-pvname: oss-pv spec: storageClassName: test capacity: storage: 20Gi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain csi: driver: ossplugin.csi.alibabacloud.com volumeHandle: oss-pv nodePublishSecretRef: name: oss-secret namespace: default volumeAttributes: bucket: "<your OSS Bucket Name>" url: "<your OSS Bucket Endpoint>" otherOpts: "-o umask=022 -o allow_other"NoteThis YAML creates a Secret and a PV. The Secret stores your AccessKey pair, which the PV uses for secure access. Replace the values of
akIdandakSecretwith your actual AccessKey ID and AccessKey Secret.The following table describes the PV parameters:
Parameter
Description
alicloud-pvnameThe label of the PV, used to bind it to a PVC.
storageClassNameThis value is for PVC binding only and does not need to correspond to an existing StorageClass object.
storageThe capacity of the OSS volume. The capacity specified for a static OSS volume is for declarative purposes only. The actual capacity is not limited, and the available capacity is based on the usage displayed in the OSS console.
accessModesThe access mode. Valid values:
ReadOnlyMany, which lets multiple pods mount the volume as read-only, andReadWriteMany, which lets multiple pods mount the volume as read-write.persistentVolumeReclaimPolicyThe reclaim policy of the PV.
driverThe driver type. Set this to
ossplugin.csi.alibabacloud.comto use the Alibaba Cloud OSS Container Storage Interface (CSI) plugin.volumeHandleThe UID (unique identifier) of the PV. This must be the same as
metadata.name.nodePublishSecretRefReferences the Secret that contains the AccessKey pair for authorization.
bucketThe name of the OSS bucket. Replace the value of
bucketwith your actual OSS bucket name.urlThe endpoint of the OSS bucket. Replace the value of
urlwith the actual endpoint of your OSS bucket.If the OSS bucket and the ACS cluster are in the same region, use the internal endpoint. For example,
oss-cn-shanghai-internal.aliyuncs.com.If the OSS bucket is region-agnostic or is in a different region from the ACS cluster, use the public endpoint. For example,
oss-cn-shanghai.aliyuncs.com.
otherOptsSpecifies custom mount options for the OSS volume in the format
-o *** -o ***. For example,-o umask=022 -o max_stat_cache_size=100000 -o allow_other.Create the Secret and the PV.
kubectl create -f oss-pv.yamlView the PV.
kubectl get pvExpected output:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE oss-pv 20Gi RWX Retain Available test <unset> 9s
Step 2: Create a PVC
Save the following YAML content as oss-pvc.yaml.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: oss-pvc
spec:
storageClassName: test
accessModes:
- ReadWriteMany
resources:
requests:
storage: 20Gi
selector:
matchLabels:
alicloud-pvname: oss-pvThe following table describes the parameters:
Parameter | Description |
| This value is for PVC binding only and does not need to correspond to an existing StorageClass object. It must be the same as the |
| The access mode. |
| The amount of storage to request. This value must not be greater than the PV's capacity. |
| The label of the PV to bind. It must be the same as the |
Create the PVC.
kubectl create -f oss-pvc.yamlView the PVC.
kubectl get pvcThe expected output shows that the PVC is bound to the PV created in Step 1.
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
oss-pvc Bound oss-pv 20Gi RWX test <unset> 6sStep 3: Create an application and mount the volume
Create a file named oss-test.yaml with the following content.
The following YAML example creates a Deployment with two pods. Both pods request storage by using the PVC named
oss-pvc, and the mount path for both is/data.
apiVersion: apps/v1
kind: Deployment
metadata:
name: oss-test
labels:
app: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: registry.cn-hangzhou.aliyuncs.com/acs-sample/nginx:latest
ports:
- containerPort: 80
volumeMounts:
- name: pvc-oss
mountPath: /data
volumes:
- name: pvc-oss
persistentVolumeClaim:
claimName: oss-pvcCreate the Deployment and mount the OSS volume.
kubectl create -f oss-test.yamlCheck the status of the pods in the Deployment.
kubectl get pod | grep oss-testThe example output indicates that two pods have been created.
oss-test-****-***a 1/1 Running 0 28s
oss-test-****-***b 1/1 Running 0 28sCheck the mount path.
The following command lists the data at the OSS bucket mount point. By default, the output is empty.
kubectl exec oss-test-****-***a -- ls /dataConsole
Step 1: Create a PV
Log on to the ACS console.
On the Clusters page, click the name of the cluster to go to the cluster management page.
-
In the left-side navigation pane of the cluster management page, choose .
-
On the Persistent Volumes page, click Create.
In the Create dialog box, configure the parameters and click the Create button.
Parameter
Description
Example
PV Type
The volume type. Select OSS.
OSS
Volume Name:
The name of the PV. Specify a custom name according to the format requirements displayed in the UI.
oss-pv
Capacity
The capacity of the OSS volume. The capacity specified for a static OSS volume is for declarative purposes only. The actual capacity is not limited, and the available capacity is based on the usage displayed in the OSS console.
20Gi
Access Mode
The access mode. Select an option based on your requirements:
ReadOnlyMany: The volume can be mounted as read-only by multiple pods.
ReadWriteMany: The volume can be mounted as read-write by multiple pods.
ReadWriteMany
Access Certificate
For security, store the AccessKey information in a Secret. This example uses the Create Secret option.
Create Secret
namespace: default
Name: oss-secret
AccessKey ID: ********
AccessKey Secret: ********
Bucket ID:
Select an OSS bucket.
oss-acs-***
OSS Path
The directory to mount. This defaults to the root directory (
/). If you specify a subdirectory, such as/dir, ensure it exists./
Endpoint:
The endpoint of the OSS bucket.
If the OSS bucket and the ACS cluster are in the same region, select the Internal Endpoint option.
If the OSS bucket is region-agnostic or is in a different region from the ACS cluster, select the Public Endpoint option.
Private Endpoint
After the PV is created, you can view its information on the Persistent Volumes page. The PV is not yet bound to a PVC.
Step 2: Create a PVC
-
In the left-side navigation pane of the cluster management page, choose .
-
On the Persistent Volume Claims page, click Create.
In the dialog box that appears, configure the parameters and click the Create button.
Parameter
Description
Example
PVC Type
The volume type. Select OSS.
OSS
Name
The name of the PVC. Specify a custom name according to the format requirements displayed in the UI.
oss-pvc
Allocation Mode
The allocation mode. Select the Existing Volumes option.
Existing PV
Existing Volumes
Select the PV that you created.
oss-pv
Capacity
The amount of storage to request. This value cannot exceed the PV's capacity.
20Gi
After the PVC is created, you can view it on the Persistent Volume Claims page. The PVC is now bound to the PV (the OSS volume).
Step 3: Create an application and mount the volume
-
In the left-side navigation pane of the cluster management page, choose .
On the Deployments page, click the Create from Image button.
Configure the parameters for the Deployment and click the Create button.
The following table describes key parameters. Keep the default values for other parameters. For more information, see Create a Deployment.
Section
Parameter
Description
Example
Basic Information
Application Name
The name of the Deployment. Specify a custom name according to the format requirements displayed in the UI.
oss-test
Basic Information
Replicas:
The number of replicas (pods) for the Deployment.
2
Container
Image Name
The image address used to deploy the application.
registry.cn-hangzhou.aliyuncs.com/acs-sample/nginx:latest
Container
Required Resources
Set the required vCPU and memory resources.
0.25 vCPU, 0.5 GiB
Volume
Add PVC
Click the button and configure the parameters:
Mount Source: Select the PVC that you created.
Container Path: Enter the container path where you want to mount the OSS bucket.
Mount source: oss-pvc
Container path: /data
Check the status of the application Deployment.
On the Deployments page, click the application name.
On the Pods tab, make sure the pods are in the Running state.
ossfs 2.0 volume
ACS supports mounting static ossfs 2.0 volumes only by using kubectl.
Step 1: Create a PV
Save the following YAML content as
oss-pv.yaml.apiVersion: v1 kind: Secret metadata: name: oss-secret namespace: default stringData: akId: <your AccessKey ID> akSecret: <your AccessKey Secret> --- apiVersion: v1 kind: PersistentVolume metadata: name: oss-pv labels: alicloud-pvname: oss-pv spec: storageClassName: test capacity: storage: 20Gi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain csi: driver: ossplugin.csi.alibabacloud.com volumeHandle: oss-pv nodePublishSecretRef: name: oss-secret namespace: default volumeAttributes: fuseType: ossfs2 # Explicitly specifies the ossfs 2.0 client bucket: "<your OSS Bucket Name>" url: "<your OSS Bucket Endpoint>" otherOpts: "-o close_to_open=false" # Note: Supported mount options are not compatible with the ossfs 1.0 client.NoteThis YAML creates a Secret and a PV. The Secret stores your AccessKey pair, which the PV uses for secure access. Replace the values of
akIdandakSecretwith your actual AccessKey ID and AccessKey Secret.The following table describes the PV parameters:
Parameter
Description
alicloud-pvnameThe label of the PV, used to bind it to a PVC.
storageClassNameThis value is for PVC binding only and does not need to correspond to an existing StorageClass object.
storageThe capacity of the OSS volume. The capacity specified for a static OSS volume is for declarative purposes only. The actual capacity is not limited, and the available capacity is based on the usage displayed in the OSS console.
accessModesThe access mode. Valid values:
ReadOnlyMany, which lets multiple pods mount the volume as read-only, andReadWriteMany, which lets multiple pods mount the volume as read-write.persistentVolumeReclaimPolicyThe reclaim policy of the PV.
driverThe driver type. Set this to
ossplugin.csi.alibabacloud.comto use the Alibaba Cloud OSS Container Storage Interface (CSI) plugin.volumeHandleThe UID (unique identifier) of the PV. This must be the same as
metadata.name.nodePublishSecretRefReferences the Secret that contains the AccessKey pair for authorization.
fuseTypeMust be
ossfs2for the ossfs 2.0 client.bucketThe name of the OSS bucket. Replace the value of
bucketwith the name of your OSS bucket.urlThe endpoint of the OSS bucket. Replace the value of
urlwith the actual endpoint of your OSS bucket.If the OSS bucket and the ACS cluster are in the same region, use the internal endpoint. For example,
oss-cn-shanghai-internal.aliyuncs.com.If the OSS bucket is region-agnostic or is in a different region from the ACS cluster, use the public endpoint. For example,
oss-cn-shanghai.aliyuncs.com.
otherOptsSpecifies custom mount options for the OSS volume in the format
-o *** -o ***. For example,-o close_to_open=false.close_to_open: Disabled by default. If you enable it, the system sends a GetObjectMeta request to OSS each time a file is opened. This retrieves the latest metadata and keeps it up to date. However, in scenarios that involve reading a large number of small files, frequent metadata queries can significantly increase latency.For more information about optional parameters, see ossfs 2.0 mount options.
Create the Secret and the PV.
kubectl create -f oss-pv.yamlView the PV.
kubectl get pvExpected output:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE oss-pv 20Gi RWX Retain Available test <unset> 9s
Step 2: Create a PVC
Save the following YAML content as oss-pvc.yaml.
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: oss-pvc spec: storageClassName: test accessModes: - ReadWriteMany resources: requests: storage: 20Gi selector: matchLabels: alicloud-pvname: oss-pvThe following table describes the parameters:
Parameter
Description
storageClassNameThis value is for PVC binding only and does not need to correspond to an existing StorageClass object. It must be the same as the
spec.storageClassNameof the PV.accessModesThe access mode.
storageThe amount of storage to request. This value must not be greater than the PV's capacity.
alicloud-pvnameThe label of the PV to bind. It must be the same as the
metadata.labels.alicloud-pvnameof the PV.Create the PVC.
kubectl create -f oss-pvc.yamlView the PVC.
kubectl get pvcThe expected output shows that the PVC is bound to the PV created in Step 1.
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE oss-pvc Bound oss-pv 20Gi RWX test <unset> 6s
Step 3: Create an application and mount the volume
Create a file named
oss-test.yamlwith the following content.The following YAML example creates a Deployment with two pods. Both pods request storage by using the PVC named
oss-pvc, and the mount path for both is/data.apiVersion: apps/v1 kind: Deployment metadata: name: oss-test labels: app: nginx spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: registry.cn-hangzhou.aliyuncs.com/acs-sample/nginx:latest ports: - containerPort: 80 volumeMounts: - name: pvc-oss mountPath: /data volumes: - name: pvc-oss persistentVolumeClaim: claimName: oss-pvcCreate the Deployment and mount the OSS volume.
kubectl create -f oss-test.yamlCheck the status of the pods in the Deployment.
kubectl get pod | grep oss-testThe example output indicates that two pods have been created.
oss-test-****-***a 1/1 Running 0 28s oss-test-****-***b 1/1 Running 0 28sCheck the mount path.
The following command lists the data at the OSS bucket mount point. By default, the output is empty.
kubectl exec oss-test-****-***a -- ls /data
Verify OSS shared storage and persistent storage
The Deployment created in the preceding example contains two pods. Both pods mount the same OSS bucket. You can verify shared storage and persistent storage as follows:
To verify shared storage, create a file in one pod and then check that it is visible in the other pod.
To verify persistent storage, restart the Deployment and then check if the original data is still accessible from a new pod.
Get the pod information.
kubectl get pod | grep oss-testSample output:
oss-test-****-***a 1/1 Running 0 40s oss-test-****-***b 1/1 Running 0 40sVerify shared storage.
Create a file in one of the pods.
This example uses the pod named
oss-test-****-***a.kubectl exec oss-test-****-***a -- touch /data/test.txtCheck for the file in the other pod.
This example uses the pod named
oss-test-****-***b.
kubectl exec oss-test-****-***b -- ls /dataThis output confirms that the
test.txtfile is shared.test.txtVerify persistent storage.
Restart the Deployment.
kubectl rollout restart deploy oss-testCheck the pod status and wait for the new pods to enter the Running state.
kubectl get pod | grep oss-testSample output:
oss-test-****-***c 1/1 Running 0 67s oss-test-****-***d 1/1 Running 0 49sIn one of the new pods, check if the data in the file system still exists.
This example uses the pod named
oss-test-****-***c.kubectl exec oss-test-****-***c -- ls /dataThis output confirms that the data in the OSS bucket persists and is accessible from the new pod's mount directory.
test.txt