Use appendable objects to support concurrent read and write-Object Storage Service(OSS)-阿里云帮助中心

Basic Principles

By default, ossfs 2.0 uses the UploadPart operation to upload large file shards and calls the CompleteMultipart operation to merge them when the file is closed. Because files with unmerged shards are not readable in OSS, you cannot read file content from an ossfs 2.0 mount point until the file is closed.

ossfs 2.0 also supports file uploads using the AppendObject operation for concurrent read and write scenarios.

Principle: Standard objects are converted to appendable objects. You can continuously append data to these objects, and the written part is readable in real time.
How to use: Add the --enable_appendable_object=true option when mounting. All objects under the mount point then use AppendObject for writes. You can also configure --appendable_object_autoswitch_threshold to enable automatic conversion to appendable objects.
Limits: The maximum size for a single file is 5 GiB. Writes will fail for files larger than this limit.
Scenarios: Concurrent read and write use cases, such as log writing.

Solution comparison

Note

In this topic, "append write" refers to the complete set of operations: opening a file, writing to the file, and closing the file. Therefore, writing data multiple times to a file after opening it is considered a single "append write".

Item	Default mode	Appendable object mode
Method to append data to a file	Call the UploadPartCopy operation for existing data in the OSS file. Call the UploadPart operation for the data to be appended. When the file is closed, call the CompleteMultipartUpload operation to complete the upload.	Call the AppendObject operation.
Frequently appending small amounts of data	Poor performance. The larger the file, the longer the OSS copy time.	Good performance. No copy overhead. Only one network request is needed.
Appending a large amount of data in a single write	Good performance. The appended data can be uploaded concurrently.	Poor performance. The appended data must be uploaded sequentially.
File size limit	None	5 GiB
Concurrent read and write	Not supported	Supported

Automatic object type conversion

By default, OSS does not support AppendObject operations on standard objects. When you mount with the --enable_appendable_object=true option, an error occurs if you append data to an existing standard object in OSS.

To handle this, ossfs 2.0 provides the --appendable_object_autoswitch_threshold option. Standard objects that do not exceed this size threshold are automatically converted to appendable objects on the first write.

Important

The AppendObject operation can only be executed sequentially, so converting a large standard object to an appendable object can take a long time. ossfs 2.0 does not perform automatic conversion by default. You must explicitly configure the mount option to enable this feature.

The automatic conversion procedure is as follows:

Rename the original object to a temporary object in OSS. The temporary object name uses the prefix .ossfs_hidden_file.
Read the object from OSS and use the AppendObject operation to upload it to the original path.
After all data is uploaded, delete the temporary object.

Important

If an exception such as a service breakdown occurs during the conversion process, you must manually recover the complete data of the file being converted. To do this, rename the corresponding temporary object.