OSS mount feature

Updated at:

MaxFrame introduces the OSS file mounting feature, which seamlessly maps OSS Buckets to the MaxCompute local file system. You can use MaxFrame's declarative configuration to define mount paths and operate directly on OSS data using POSIX-based APIs. This eliminates the need for traditional SDK development and significantly reduces development complexity.

Use cases

  • Large-scale data processing: Suitable for offline computing tasks that require frequent interaction between OSS and MaxCompute.

  • Data pre-processing: Read raw data from OSS, and then clean, transform, and write the data to MaxCompute.

  • Hybrid storage architecture: Store data in OSS while enabling efficient collaboration with MaxCompute.

  • Version requirements: Requires MaxFrame SDK version 2.4.0 or later.

Core features

  • Secure authentication

    Supports authentication based on Resource Access Management (RAM) Roles. This method avoids exposing long-term AccessKey pairs and improves data access security.

  • Seamless integration

    • Maps an OSS Bucket to a MaxCompute local file system path. Supports direct read and write operations.

    • Compatible with POSIX-based APIs, such as open, read, and write. This simplifies file management.

  • Declarative configuration

    You can use the MaxFrame declarative configuration decorator @with_fs_mount to define mount paths. This completes the data mapping without requiring complex code.

  • Improved development efficiency

    SDK coding is not required. Developers can operate directly on OSS data through APIs, which shortens the development cycle.

Core parameters

path - Remote storage address definition

  • Format: scheme://<endpoint>/<bucket>/<object_prefix>

    • scheme: The protocol type (OSS).

    • endpoint: The access endpoint for the storage service.

    • bucket: The bucket name.

    • object_prefix: The object prefix path within the bucket.

  • Example:

    path = "oss://oss-cn-shanghai-internal.aliyuncs.com/maxframe-test/oss_data"

mount_path - MaxCompute local mount target

  • Description: The mount_path parameter specifies the local access path in MaxCompute for the remote storage. MaxFrame automatically creates this directory and mounts the file system.

  • Configuration requirements:

    • Permissions: The user who runs the task must have read and write permission for the directory.

    • Uniqueness: Each mount target is independent. Avoid path conflicts.

    • Lifecycle: Automatically uninstalls in an exception-safe manner after the decorator finishes execution.

  • Example:

    mount_path = "/mnt/oss_data"  # MaxCompute local mount path

storage_options - Authentication and configuration parameters

Authentication using a RAM Role

  1. Log in to the RAM console.

  2. In the left navigation bar, select Identities > Roles.

  3. In the Basic Information section, you can obtain the ARN.

  4. Example:

    storage_options = {
      "role_arn": "acs:ram::xxxxxxxx:role/MaxFrameAccessRole"
    }

    Example RAM Policy that follows the principle of least privilege for read-only access:

    {
        "Statement": [
            {
                "Action": "oss:GetObject",
                "Effect": "Allow",
                "Resource": "acs:oss:*:*:maxframe-***-poc/oss_data/*"
            }
        ],
        "Version": "1"
    }

Examples

The following code snippets provide examples of how to use this feature. For complete code examples, see OSS mounting and best practices.

OSS data processing

from maxframe.udf import with_fs_mount

@with_running_options(engine="dpe", cpu=2, memory=8)
@with_fs_mount(
    path="oss://oss-cn-shanghai-internal.aliyuncs.com/bigdata-bucket/preprocessed/",
    mount_path="/mnt/data",
    storage_options={"role_arn": "acs:ram::123456:role/DataAccess"}
)
def etl_job():
    df = pd.read_csv("/mnt/data/2025-*.csv")
    # Distributed computing...

Loading OSS model files

from maxframe.udf import with_fs_mount

@with_running_options(engine="dpe", cpu=2, memory=8)
@with_fs_mount(
    path="oss://oss-cn-shanghai-internal.aliyuncs.com/model-registry/model_path",
    mount_path="/mnt/model_path",
    storage_options={"role_arn": "acs:ram::123456:role/ModelAccess"}
)
def resume_training():
    model = load_model("/mnt/model_path/model.pkl")
    # Model training...

Add OSS mount dependencies to a custom image

The built-in MaxFrame base images include the FUSE libraries that OSSFS requires, so the mount feature works directly through with_fs_mount. When a custom image is used, make sure that the required dependencies are installed in the image.

  1. Download

    When building a custom image from an Ubuntu base image, install the OSS FUSE dependency in the image: wget http://odps-oss-test.oss-test.aliyun-inc.com/maxframe-build-dependency/ossfs2_2.0.3.1_linux_x86_64.deb.

  2. Install

    Local installation

    COPY ossfs2_2.0.3.1_linux_x86_64.deb /tmp/ossfs2_2.0.3.1_linux_x86_64.deb
    RUN dpkg -i /tmp/ossfs2_2.0.3.1_linux_x86_64.deb && \
        ossfs2 --version && \
        rm -rf /tmp/ossfs2_2.0.3.1_linux_x86_64.deb

    Installation from OSS

    To install from OSS, upload the downloaded package to OSS first, and then run the following command.

    RUN wget http://odps-oss-test.oss-test.aliyun-inc.com/maxframe-build-dependency/ossfs2_2.0.3.1_linux_x86_64.deb && \
        dpkg -i ossfs2_2.0.3.1_linux_x86_64.deb && \
        ossfs2 --version && \
        rm -rf ossfs2_2.0.3.1_linux_x86_64.deb