Accelerate OSS access by using EFC

Runtime environment

Hardware resource requirements

The EFC client consumes memory and disk resources on compute nodes. You must plan your resources based on whether you enable the cache feature.

• The cache is disabled

  • Memory: Resident memory usage is typically less than 1 GiB. You must reserve at least 5 GiB of available memory to handle burst workloads.

  • Disk: No special disk space is required. Ensure that the system disk has enough space to store operational logs.

• Enable the cache feature

  When the cache is enabled, resource consumption consists mainly of the cache medium and index overhead.

  • Memory resources

    • Base consumption: Same as when the cache feature is disabled.

    • Memory cache: If you configure memory as the cache, an amount of memory equal to the configured size is consumed.

    • Index overhead: Regardless of whether you use a memory or disk cache, additional memory is required to maintain the data index. You must reserve an amount of memory equal to approximately 0.1% of the total cache capacity.

    • Formula: Total memory consumption ≈ Base runtime memory + Memory cache size + (Memory cache size + Disk cache size) × 0.001

  • Disk resources:

    • Disk cache: If you configure a disk as the cache medium, it consumes an amount of disk space equal to the configured size. For example, a 1 TiB disk cache configuration consumes 1 TiB of disk space.

Feature limitations

• Access mode: Only read-only access to OSS data is supported. After mounting, the owner of all files is set to root with read-only permissions. The chmod and chown operations are not supported.

• POSIX compatibility: Supports operations such as open, close, read, readdir, readdirplus, lookup, and getattr. Does not support operations such as readlink, write, rename, setattr, link, or symlink.

• Storage class: Only access to objects in the Standard and Infrequent Access storage classes is supported. Direct access to objects in the Archive, Cold Archive, or Deep Cold Archive storage classes is not supported.

• Path compatibility:

  • Path conflicts: If an OSS Bucket contains both a file named a/b and a folder named a/b/, only the folder a/b/ can be accessed after mounting.

  • Special paths: You cannot access objects whose ObjectKey starts with /, contains consecutive //, or contains . or ...

• Access permissions: Mounting and accessing OSS requires at least the oss:GetBucketStat, oss:ListObjects, and oss:GetObject permissions.

• Data high availability: EFC is a read-only cache and does not guarantee data high availability. Cached data may be lost due to events such as a hardware failure or node replacement, which invalidates the cache. When the corresponding data is accessed again, an origin fetch is triggered. This can cause performance fluctuations.

Configuration 1: Basic mount (quick verification)

You can use this configuration to quickly verify that EFC can access your OSS Bucket. This configuration does not enable cache acceleration.

Mount using an STS credential (Recommended):

# Create a local mount target
mkdir -p /mnt/oss_data

# Execute the mount (replace with your Bucket domain name)
mount -t alinas \
  -o efc,protocol=oss,g_oss_STSFile=/etc/passwd-sts \
  your-bucket.oss-cn-hangzhou.aliyuncs.com:/ /mnt/oss_data

Mount using an AccessKey:

# Create a local mount target
mkdir -p /mnt/oss_data

# Execute the mount (replace with your Bucket domain name)
mount -t alinas \
  -o efc,protocol=oss,passwd_file=/etc/passwd-oss \
  your-bucket.oss-cn-hangzhou.aliyuncs.com:/ /mnt/oss_data

Command description:

• mount -t alinas: Mounts the file system using the EFC client (alinas type).

• your-bucket.oss-cn-hangzhou.aliyuncs.com:/ /mnt/oss_data: The mount target configuration. your-bucket.oss-cn-hangzhou.aliyuncs.com is your OSS Bucket domain name, and /mnt/oss_data is the local mount target.

• g_oss_STSFile=/etc/passwd-sts: Specifies the path to the configuration file that contains the STS credentials.

• passwd_file=/etc/passwd-oss: Specifies the path to the password file that contains the OSS AccessKey.

For more information, see Detailed mount parameters.

Verify the mount result:

# Check if the mount was successful
df -h | grep /mnt/oss_data
# List the files in OSS
ls /mnt/oss_data

If you can view the files in your OSS Bucket, the basic mount is successful. You can now enable cache acceleration as needed.

Configuration 2: Single-node cache mode

You can enable the memory and disk cache on the current node. This configuration builds on the basic mount and is suitable for single-node tasks or quick tests.

Note: If you have already completed the basic mount, you must first unmount it by running the umount /mnt/oss_data

# Create the disk cache folder
mkdir -p /mnt/cache/

# Mount and enable single-node cache (using an STS credential)
mount -t alinas \
  -o efc,protocol=oss,g_oss_STSFile=/etc/passwd-sts \
  -o g_tier_DadiIsDistributed=false \
  -o g_tier_DadiMemCacheCapacityMB=1024 \
  -o g_tier_DadiDiskCacheCapacityMB=10240 \
  -o g_tier_DadiDiskCachePath=/mnt/cache/ \
  -o g_server_Port=17871 \
  your-bucket.oss-cn-hangzhou.aliyuncs.com:/ /mnt/oss_data

Command description:

• mount -t alinas ... your-bucket.oss-cn-hangzhou.aliyuncs.com:/ /mnt/oss_data: Mounts the OSS Bucket to the local /mnt/oss_data folder using the EFC client.

• g_oss_STSFile=/etc/passwd-sts: Specifies the path to the configuration file that contains the STS credential.

• g_tier_DadiIsDistributed=false: Single-node cache mode. The node does not communicate with other nodes.

• g_tier_DadiMemCacheCapacityMB=1024: Allocates 1 GB of memory for data caching.

• g_tier_DadiDiskCacheCapacityMB=10240: Allocates 10 GB of disk space for data caching.

• g_tier_DadiDiskCachePath=/mnt/cache/: The folder where disk cache files are stored.

• g_server_Port=17871: Opens port 17871. This lets you use command-line tools to view the cache hit ratio or perform data prefetching.

For more information, see Detailed mount parameters.

Configuration 3: Distributed cache cluster mode

This mode combines multiple nodes into a cache cluster to share cached data and accelerate data distribution over a P2P network. This mode is suitable for large-scale training and inference in a production environment.

1. Configure the cache node list

On all nodes, create the /etc/efc/rootlist file and add the information for all nodes in the cluster:

# Format: <Unique_ID>:<Node_Internal_IP>:<P2P_Port>
1:192.168.1.1:17980
2:192.168.1.2:17980
3:192.168.1.3:17980

Warning: The rootlist file must be identical across the entire cluster. Inconsistencies can prevent some nodes from joining the cache network and can lower the cache hit ratio.

2. Run the mount command on all nodes

# Create the disk cache folder
mkdir -p /mnt/cache/

# Mount and enable distributed cache (using an STS credential)
mount -t alinas \
  -o efc,protocol=oss,g_oss_STSFile=/etc/passwd-sts \
  -o g_tier_EnableClusterCache=true \
  -o g_tier_DadiIsDistributed=true \
  -o g_tier_DadiAddr=/etc/efc/rootlist \
  -o g_tier_DadiMemCacheCapacityMB=1024 \
  -o g_tier_DadiDiskCacheCapacityMB=10240 \
  -o g_tier_DadiDiskCachePath=/mnt/cache/ \
  -o g_server_Port=17871 \
  your-bucket.oss-cn-hangzhou.aliyuncs.com:/ /mnt/oss_data

Command description:

• mount -t alinas ... your-bucket.oss-cn-hangzhou.aliyuncs.com:/ /mnt/oss_data: Mounts the OSS Bucket to the local /mnt/oss_data folder using the EFC client.

• g_oss_STSFile=/etc/passwd-sts: Specifies the path to the configuration file that contains the STS credential.

• g_tier_EnableClusterCache=true, g_tier_DadiIsDistributed=true: Enables the core feature of EFC: distributed cache acceleration.

• g_tier_DadiAddr=/etc/efc/rootlist: Specifies the path to the cluster node configuration file, which is used to build the P2P cache network.

• g_tier_DadiMemCacheCapacityMB=1024: Allocates 1 GB of memory for data caching.

• g_tier_DadiDiskCacheCapacityMB=10240: Allocates 10 GB of disk space for data caching. The cache files are stored in the /mnt/cache/ folder.

• g_server_Port=17871: Opens port 17871. This lets you use command-line tools to view the cache hit ratio or perform data prefetching.

For more information, see Detailed mount parameters.

Data prefetching

This feature loads data from OSS into the cache in advance to eliminate the latency of the first access. This is especially important for scenarios such as model inference.

• Path requirement: The prefetch path must be a relative path within the mount target.

• Task status: running, completed, canceled, or failed.

• Passive task eviction: Prefetch tasks are not retained indefinitely. The system automatically evicts historical task records based on the following rules:

  • Tasks in the running state: Can be canceled only using the cancel command and cannot be passively evicted.

  • Tasks that are completed, canceled, or have failed are automatically evicted if either of the following conditions is met:

    • The total number of historical prefetch tasks exceeds 10,000.

    • The task record is more than 7 days old.

Cache monitoring

This feature lets you view the real-time running status of the cache, such as the hit rate and throughput. This information is key for performance tuning.

• View monitoring data

  • HTTP method (Recommended, returns JSON format)

    curl -s "http://localhost:17871/v1/tier"

  • Command-line method (Returns text format)

    aliyun-alinas-efc-cli -m <mount_target> -r tier

• Clear monitoring data

  Reset monitoring counters (such as tier_read_hit) to zero.

  • HTTP method

    curl -s "http://localhost:17871/v1/tier/clear"

  • Command-line method

    aliyun-alinas-efc-cli -m <mount_target> -r tier -c

• Key metric descriptions

  The following metrics describe the possible paths when Client A accesses the cache of Client B. In this scenario, Client B provides the cache service:

  

  Metric name	Description	Corresponding path
  tier_read	Total number of read requests that passed through the cache path.	i
  tier_read_bytes	Total data volume of read requests that passed through the cache path.	i
  tier_read_hit	Number of successful read requests from a distributed node that resulted in a cache hit.	ii
  tier_read_hit_bytes	Total data volume of successful read requests from a distributed node that resulted in a cache hit (may include read amplification).	ii
  tier_read_miss	Number of successful read requests from a distributed node that resulted in a cache miss (origin fetch). This triggers the distributed node to read from the backend.	iii
  tier_read_miss_bytes	Total data volume of successful read requests from a distributed node that resulted in a cache miss.	iii
  tier_direct_read	Number of requests where a read from a distributed node failed, and the client successfully read directly from the backend OSS.	v
  tier_direct_read_bytes	Total data volume of requests where a read from a distributed node failed, and the client successfully read directly from the backend OSS.	v
  tier_root_read_source	Number of requests to the backend when the client acts as a distributed node and a cache miss occurs.	iv
  tier_root_read_source_bytes	Total data volume read from the backend when the client acts as a distributed node and a cache miss occurs.	iv

Scaling the cache

Add or remove nodes online

This operation dynamically adjusts the number of nodes in the distributed cache cluster without service interruption.

• Scale-out: In the rootlist file on all nodes, add the IP addresses of the new nodes.

• Scale-in: In the rootlist file on all nodes, delete the IP addresses of the target nodes.

• Effective mechanism: The EFC process periodically rereads the rootlist file every 5 seconds by default and automatically applies the changes.

Adjust the cache capacity of a single node

Important: EFC does not support dynamic adjustment of the cache capacity of a single node. To make adjustments, you must unmount and remount the file system. We recommend that you perform this operation during off-peak hours.

When you remount, you can modify the following parameters to adjust the cache capacity:

Dynamically adjust mount parameters

EFC provides a Python script tool that you can use to modify mount parameters without manually unmounting the file system. The tool works by modifying the EFC state file, which stores the start command, and then killing the current EFC process. The process automatically loads the new parameters upon restart.

• Download the tool

  wget https://aliyun-alinas-eac-ap-southeast-1.oss-ap-southeast-1.aliyuncs.com/cache/efc_runtime_ops.latest.py

• Modify mount parameters

  python3 efc_runtime_ops.py update <local_mount_path> [-o <update_parameters,...>]...

  Example:

  python3 efc_runtime_ops.py update /mnt/oss_data \
            -o g_tier_EnableClusterCache=true,g_tier_tier_DadiP2PPort=23456 \
            -o g_tier_BlockSize=1048576

• Roll back

  When you modify parameters, the tool automatically backs up the old state file. If EFC fails after the modification, you can roll back to recover the previous state:

  python3 efc_runtime_ops.py rollback <local_mount_path> [source_path]

  Example: Restore the state file using the default backup file path.

  python3 efc_runtime_ops.py rollback /mnt/oss_data

Directory cache management

EFC enables directory caching by default to accelerate directory traversal operations, such as ls. If the directory structure on the OSS side changes and EFC does not detect the change promptly, you can manually clear the directory cache.

Directory cache parameters (Specified at mount time):

Clear the directory cache:

• HTTP method

  curl -s "http://localhost:17871/v1/action/clear_readdir_cache"

• Command-line method

  aliyun-alinas-efc-cli -m <mount_target> -r action -k clear_readdir_cache

Basic parameters

Cache configuration parameters

Metadata and directory cache parameters

O&M and advanced parameters

Permissions and security

• Protect the password file: Set the permissions for the password file to 600 and store it in a secure location to prevent credential leaks.

• Prioritize STS: For security reasons, use STS temporary credentials instead of long-term AccessKeys to reduce the risk of key leakage.

• Minimize security group exposure: Open the peer-to-peer communication port 17980 only between cluster nodes and strictly limit the access sources for the management port 17871.

Scenario-specific configuration recommendations

• AI model training

  • Characteristics: This scenario typically involves repeated, sequential reads of large datasets, which may contain many small files.

  • Recommendations:

    • Configure a disk cache that is large enough to hold the entire hot spot dataset, or most of it.

    • Before the training task starts, perform data prefetching on the core dataset to handle the overhead of the first read in advance.

• AI model inference

  • Characteristics: In this scenario, many compute nodes need to load the same model files simultaneously.

  • Recommendations:

    • Enable the distributed cluster mode to fully use the P2P network to distribute model files. This avoids creating a single point of pressure on OSS.

    • When the service goes online or when you update a model, perform data prefetching for the model files.

Data consistency management

• Understand eventual consistency: EFC is a read-only cache. Changes to the OSS backend are not synchronized in real time. Therefore, your application design must tolerate a certain degree of data latency.

• Proactive refresh: For scenarios that require frequent updates and immediate reads, you can add a step in your application logic to call the interface for clearing the directory cache.

• Enable metadata caching: For scenarios in which metadata does not change often, you can enable metadata caching using -o g_metadata_CacheEnable. This can reduce the number of GetObjectMeta requests to OSS.