Speed up the ls command for optimized listing performance-Object Storage Service(OSS)-阿里云帮助中心

Prerequisites

An Object Storage Service (OSS) bucket is created.
ossfs is installed.

Scenarios

The bucket contains a large number of objects or directories.
The bucket does not contain any existing symbolic links, and you do not intend to use the symbolic link feature later. Additionally, you do not require real-time updates of metadata such as the user ID (uid) and group ID (gid).
Metadata consistency is not a primary concern in, for example, model training or inference scenario. In these cases, files added to the dataset typically remain unchanged.

Parameters

Parameter	Description
max_stat_cache_size	Specifies the maximum number of objects and directories whose metadata can be cached. In V1.91.x, this parameter defaults to 100000. The default setting requires approximately 40 MB of memory. In earlier versions, this parameter defaults to 1000. We recommend that you configure this parameter based on the number of frequently accessed objects in the bucket and the memory size. If the memory is sufficient, set this parameter to a reasonably large value (for example, 1000000) to have optimal performance. If you set this parameter to 0, metadata cache is disabled. However, performance will be impacted because of large amount of OSS metadata queries. Note If you want to retrieve the most recent metadata, you can leave this parameter empty. ossfs automatically retrieves the most recent metadata when handling open requests with POSIX semantics. If you set this parameter to 0, the number of HeadObj requests in ossfs may far exceed that of read/write operations during high-frequency read/write operations. After setting the parameter to the default value (100000), the number of HeadObj requests decreases to align with read/write operations, which greatly improves the performance. If you set this parameter to a value greater than 0, metadata cache is enabled and the specified value is the maximum number of objects and directories whose metadata can be cached.
readdir_optimize	Specifies whether to use cache optimization. This parameter defaults to false. After you specify the parameter, ossfs does not send a HeadObject request to obtain the object metadata, such as `gid` and `uid`, when you run the `ls` command. You should configure this parameter based on your application requirements. However, even if you set this parameter to false, a certain amount of HeadObject requests are still sent for reasons such as permission checks.
stat_cache_expire	Specifies whether to enable cache expiration or the cache validity period in seconds if cache expiration is enabled. This parameter defaults to 900 (15 minutes). If you set this parameter to -1, cache expiration is disabled. We recommend that you do not disable cache expiration unless you are certain that no data updates will occur after the bucket is mounted. If you set this parameter to a value greater than 0, cache expiration is enabled and the specified value is the cache validity period in seconds.

Mounting examples

Scenario 1: Mount a bucket that contains a large number of objects or directories

The following sample command mounts a bucket named mybucket at the local /mnt/oss directory and caches the metadata of a maximum of 1,000,000 objects and directories (approximately 400 MB of memory).
```
ossfs mybucket /mnt/oss -ourl=http://oss-cn-****.aliyuncs.com -omax_stat_cache_size=1000000 
```
Scenario 2: Mount a bucket that contains no symbolic links in an application that does not involve the use of symbolic links later and does not require real-time updates of metadata such as the user ID (uid) and group ID (gid)

The following sample command mounts a bucket named mybucket at the local /mnt/oss directory and disables sending HeadObject requests in the ls command.
```
ossfs mybucket /mnt/oss -ourl=http://oss-cn-****.aliyuncs.com -oreaddir_optimize
```
Scenario 3: Mount a bucket in a model training or inference scenario where metadata consistency is not a primary concern

The following sample command mounts a bucket named mybucket at the local /mnt/oss directory with cache expiration disabled (stat_cache_expire set to -1). You can enable cache expiration by setting the stat_cache_expire parameter to a value greater than 0, for example, 72000 (cache expired in 20 hours).
```
ossfs mybucket /mnt/oss -ourl=http://oss-cn-****.aliyuncs.com -ostat_cache_expire=-1
```