OSS mount feature
MaxFrame introduces the OSS file mounting feature, which seamlessly maps OSS Buckets to the MaxCompute local file system. You can use MaxFrame's declarative configuration to define mount paths and operate directly on OSS data using POSIX-based APIs. This eliminates the need for traditional SDK development and significantly reduces development complexity.
Use cases
Large-scale data processing: Suitable for offline computing tasks that require frequent interaction between OSS and MaxCompute.
Data pre-processing: Read raw data from OSS, and then clean, transform, and write the data to MaxCompute.
Hybrid storage architecture: Store data in OSS while enabling efficient collaboration with MaxCompute.
Version requirements: Requires MaxFrame SDK version 2.4.0 or later.
Core features
Secure authentication
Supports authentication based on Resource Access Management (RAM) Roles. This method avoids exposing long-term AccessKey pairs and improves data access security.
Seamless integration
Maps an OSS Bucket to a MaxCompute local file system path. Supports direct read and write operations.
Compatible with POSIX-based APIs, such as
open,read, andwrite. This simplifies file management.
Declarative configuration
You can use the MaxFrame declarative configuration decorator
@with_fs_mountto define mount paths. This completes the data mapping without requiring complex code.Improved development efficiency
SDK coding is not required. Developers can operate directly on OSS data through APIs, which shortens the development cycle.
Core parameters
path - Remote storage address definition
Format:
scheme://<endpoint>/<bucket>/<object_prefix>scheme: The protocol type (OSS).
endpoint: The access endpoint for the storage service.
bucket: The bucket name.
object_prefix: The object prefix path within the bucket.
Example:
path = "oss://oss-cn-shanghai-internal.aliyuncs.com/maxframe-test/oss_data"
mount_path - MaxCompute local mount target
Description: The
mount_pathparameter specifies the local access path in MaxCompute for the remote storage. MaxFrame automatically creates this directory and mounts the file system.Configuration requirements:
Permissions: The user who runs the task must have read and write permission for the directory.
Uniqueness: Each mount target is independent. Avoid path conflicts.
Lifecycle: Automatically uninstalls in an exception-safe manner after the decorator finishes execution.
Example:
mount_path = "/mnt/oss_data" # MaxCompute local mount path
storage_options - Authentication and configuration parameters
Authentication using a RAM Role
Log in to the RAM console.
In the left navigation bar, select .
In the Basic Information section, you can obtain the ARN.
Example:
storage_options = { "role_arn": "acs:ram::xxxxxxxx:role/MaxFrameAccessRole" }Example RAM Policy that follows the principle of least privilege for read-only access:
{ "Statement": [ { "Action": "oss:GetObject", "Effect": "Allow", "Resource": "acs:oss:*:*:maxframe-***-poc/oss_data/*" } ], "Version": "1" }
Examples
The following code snippets provide examples of how to use this feature. For complete code examples, see OSS mounting and best practices.
OSS data processing
from maxframe.udf import with_fs_mount
@with_running_options(engine="dpe", cpu=2, memory=8)
@with_fs_mount(
path="oss://oss-cn-shanghai-internal.aliyuncs.com/bigdata-bucket/preprocessed/",
mount_path="/mnt/data",
storage_options={"role_arn": "acs:ram::123456:role/DataAccess"}
)
def etl_job():
df = pd.read_csv("/mnt/data/2025-*.csv")
# Distributed computing...Loading OSS model files
from maxframe.udf import with_fs_mount
@with_running_options(engine="dpe", cpu=2, memory=8)
@with_fs_mount(
path="oss://oss-cn-shanghai-internal.aliyuncs.com/model-registry/model_path",
mount_path="/mnt/model_path",
storage_options={"role_arn": "acs:ram::123456:role/ModelAccess"}
)
def resume_training():
model = load_model("/mnt/model_path/model.pkl")
# Model training...Add OSS mount dependencies to a custom image
The built-in MaxFrame base images include the FUSE libraries that OSSFS requires, so the mount feature works directly through with_fs_mount. When a custom image is used, make sure that the required dependencies are installed in the image.
Download
When building a custom image from an Ubuntu base image, install the OSS FUSE dependency in the image:
wget http://odps-oss-test.oss-test.aliyun-inc.com/maxframe-build-dependency/ossfs2_2.0.3.1_linux_x86_64.deb.Install
Local installation
COPY ossfs2_2.0.3.1_linux_x86_64.deb /tmp/ossfs2_2.0.3.1_linux_x86_64.deb RUN dpkg -i /tmp/ossfs2_2.0.3.1_linux_x86_64.deb && \ ossfs2 --version && \ rm -rf /tmp/ossfs2_2.0.3.1_linux_x86_64.debInstallation from OSS
To install from OSS, upload the downloaded package to OSS first, and then run the following command.
RUN wget http://odps-oss-test.oss-test.aliyun-inc.com/maxframe-build-dependency/ossfs2_2.0.3.1_linux_x86_64.deb && \ dpkg -i ossfs2_2.0.3.1_linux_x86_64.deb && \ ossfs2 --version && \ rm -rf ossfs2_2.0.3.1_linux_x86_64.deb