DocsICP FilingConsole
官方文档
首页

Lake storage

更新时间：
复制 MD 格式
我的收藏

The AnalyticDB for MySQL lake storage feature simplifies data lake construction and storage management. It supports storing structured table data in formats like Iceberg and Paimon, and unstructured file objects, making it ideal for scenarios that require both offline batch processing and real-time analytics. This topic describes the storage modes, storage types, and resource management operations for lake storage.

Prerequisites

An AnalyticDB for MySQL Enterprise Edition, Basic Edition, or Data Lakehouse Edition cluster is created.

Enable lake storage

The lake storage feature is in invitational preview. To enable this feature, submit a ticket to contact technical support.

Storage modes

Before you create a data lake table, you must decide where to store your data. AnalyticDB for MySQL offers two storage modes. The mode is determined at table creation time and cannot be changed afterward.

Dimension

Managed lake storage (internal lake)

User-owned OSS (external lake)

Storage management

AnalyticDB for MySQL fully manages and automatically handles underlying storage buckets

Users manage their own OSS buckets

Key table creation parameters

catalog_type='ADB' + adb_lake_bucket (XIHE SQL); or TBLPROPERTIES ('adb_lake_bucket' = '...') + SET spark.adb.lakehouse.enabled=true (Spark SQL)

LOCATION 'oss://...'

Enablement

Requires lake storage to be enabled first

No additional setup required for same-account access

Use cases

New projects that require simplified operations

Existing OSS data or self-managed storage requirements

  • Managed lake storage: Data is stored in underlying buckets automatically managed by AnalyticDB for MySQL. You can read from and write to lake tables using standard SQL without managing file systems, permission configurations, or lifecycle policies. For more information, see Create a lake storage.

  • User-owned OSS: Data is stored entirely in an OSS bucket that you specify in the same region. You specify the OSS path by using the LOCATION parameter when creating a table.

Note

The table creation syntax varies depending on the engine. For more information, see the read and write documentation for the corresponding engine in Next steps.

Storage types

AnalyticDB for MySQL managed lake storage offers two storage types. You can choose the appropriate storage solution based on your business requirements.

Dimension

Basic

Standard

Underlying storage

OSS object storage

AFS (AnalyticDB Filesystem) high-performance object storage

Access protocol

OSS API

S3-compatible protocol, with additional POSIX interface support (in invitational preview)

Use cases

General-purpose data analytics and offline batch processing

AI/ML training and inference, high-performance computing, and multi-modal data processing

Performance characteristics

Standard throughput and latency

High throughput, low latency, with built-in multi-level caching

Hot-cold tiering

Not supported

Supported — hot data is stored on AFS, and cold data is automatically moved to OSS

Enablement

Submit a ticket

Submit a ticket

Basic (OSS standard lake storage)

Basic lake storage is built on Alibaba Cloud OSS object storage, making it suitable for most general-purpose data analytics and offline batch processing scenarios. After creation, the system automatically creates a bucket in OSS under the service account. This bucket is in the same region as your cluster and has the same name as the lake storage.

Standard (AFS high-performance lake storage)

AFS (AnalyticDB Filesystem) is an S3-compatible, high-performance object storage that provides high-performance data ingestion and persistence capabilities for AnalyticDB for MySQL lake storage. AFS is designed for scenarios that demand higher storage performance, such as AI/ML training and inference, and large-scale multi-modal data processing.

Core capabilities:

  • S3 protocol compatibility: You can directly use standard toolchains such as AWS SDK and Spark/Flink S3A to access AFS without modifying existing tools.

  • POSIX interface (in invitational preview): The same data can be accessed through both S3 APIs and POSIX mounts, covering scenarios such as training and inference that depend on file paths.

  • Hot-cold tiering: Hot data is stored on AFS to ensure high-performance access, while cold data is automatically moved to OSS. The unified namespace is transparent to upper-layer applications, achieving a balance between cost and performance.

  • Security and permissions: AFS supports a fine-grained permission model based on S3 policies, allowing minimum-privilege authorization by bucket, path, and operation type. It also supports the user and AccessKey system for multi-team isolation.

POSIX interface

AI/ML training and inference pipelines typically depend on POSIX file paths rather than S3 keys. AFS provides a POSIX interface on top of the S3 API, offering the following key benefits:

  • Zero-modification training and inference: Ray operators and model checkpoints can directly read from and write to mount points without switching to the S3 SDK.

  • Dual-mode access to the same data: Upstream production pipelines write through S3, while downstream training or retrieval reads through the POSIX interface — zero-copy, zero-data-movement.

Important

The POSIX interface is currently in invitational preview. To use this feature, submit a ticket.

Billing

Usage notes

  • An Alibaba Cloud account can create up to five lake storages in the same region.

  • There is a delay in updating the displayed storage usage for lake storage, so you may not see the updated data volume immediately after writing data.

  • Before you delete a lake storage, you must first delete all data in it. Otherwise, the deletion fails.

  • After you create a lake storage, AnalyticDB for MySQL automatically creates a bucket in OSS under its service account. This bucket is in the same region as your AnalyticDB for MySQL cluster and has the same name as the lake storage. You can view this bucket in the OSS console under your Alibaba Cloud account by adding a favorite path.

    Click to view how to find the OSS bucket

    1. Log on to the OSS console.

    2. In the navigation pane on the left, click the Add icon next to Favorite Paths.

    3. In the Add Favorite Paths dialog box, select an Adding Method and specify the Region and Bucket Name name.

      Adding Method: Select Add from other authorized buckets.

      Region: The region of your AnalyticDB for MySQL cluster.

      Bucket: Enter the Lake Storage Name.

  • The AnalyticDB for MySQL backup and recovery feature does not support data in lake storage.

Create a lake storage

Standard

Standard lake storage is built on AFS high-performance object storage. To create a standard lake storage, you must first create a storage cluster and then create a bucket.

  1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.

  2. In the left-side navigation pane, choose Data Management > Lake Storages.

  3. Click the Storage Cluster Management tab.

  4. Click Create Cluster.

  5. In the Create Cluster dialog box, Storage Class is fixed to Standard. Set Reserved Resource Size, optionally enter a Cluster Description, and then click OK.

    Note

    It takes some time to create a storage cluster. Wait until the cluster is created before you proceed.

  6. Click the Bucket Management tab.

  7. Click Create Bucket.

  8. In the Create Bucket dialog box, configure the following parameters and click OK.

    • Type: Select Standard.

    • Cluster: Select the storage cluster created in the previous step.

    • Bucket Name: Enter a bucket name. The name must be globally unique and cannot be changed after creation.

    • Data Redundancy Type: Select Zone-Redundant Storage or Locally Redundant Storage.

    • Bucket Description: (Optional) Enter a description to distinguish between different business scenarios.

Basic

  1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.

  2. In the left-side navigation pane, choose Data Management > Lake Storages.

  3. On the Bucket Management tab, click Create Bucket.

  4. In the Create Bucket dialog box, configure the following parameters and click OK.

    • Type: Select Basic.

    • Bucket Name: Automatically generated by the system. You do not need to enter a name.

    • Bucket Description: (Optional) Enter a description to distinguish between different business scenarios.

    • Data Redundancy Type: Select Zone-Redundant Storage or Locally Redundant Storage.

    Important

    The bucket name is automatically generated in the adb-lake-<Region ID>-<Random String> format. The name is globally unique and cannot be changed after creation.

  5. (Optional) Modify the bucket description.

    The bucket name is automatically generated and cannot be modified. We recommend modifying the bucket description to distinguish it from others used for different business scenarios.

    1. In the Bucket Description column of the target bucket, click the image icon.

    2. In the Edit Bucket Description dialog box, enter a description and click OK.

  6. (Optional) Add authorization.

    In the Actions column, click Add Authorization and choose to authorize a RAM user or a RAM role.

    Note

    • Read-only: Allows users to view lake storage data, but not modify or delete it.

    • Read/write: Allows users to read, write, and modify data.

    • Only an Alibaba Cloud account or a RAM user with the required permissions can grant permissions.

    • Permission changes take effect immediately. Confirm the authorization scope before you proceed.

View connection endpoint

After you create a standard lake storage, you can view its S3 connection endpoint in the console. Use this endpoint to configure S3-compatible tools such as AWS SDK and Spark/Flink S3A to access AFS lake storage.

  1. In the navigation pane on the left, choose Data Management > Lake Storages.

  2. On the Bucket Management tab, locate the target standard bucket and view its S3 connection endpoint in the Endpoint column.

Note

This endpoint is accessible only within the same VPC. Make sure your tools or applications are deployed in the same VPC.

Basic lake storage does not support the S3 protocol. The Endpoint column displays a hyphen (-) for basic buckets.

View lake storage data volume

  1. In the navigation pane on the left, choose Data Management > Lake Storages.

  2. View the data volume in the Storage Usage column of the target lake storage.

    Important

    There is a delay in updating the displayed storage usage for lake storage, so you may not see the updated data volume immediately after writing data.

Delete a lake storage

  1. In the navigation pane on the left, choose Data Management > Lake Storages.

  2. In the Actions column of the target lake storage, click Delete.

  3. In the Delete dialog box that appears, click OK.

    Important

    Before you delete a lake storage, you must first delete all data in it. Otherwise, the deletion fails.

Next steps

Set default table format (Spark SQL only)

Important

This feature applies only to the Spark SQL engine.

You can set a default table format at the database level so that new tables created in the database automatically use the specified format without requiring a per-table declaration.

  1. In the Spark job resource group, interactive resource group, or submitted job, set the following parameter:

    spark.sql.adb.sources.extractProviderFromDBProperties.enabled true

  2. When creating a database, specify DBPROPERTIES with 'storage.format' set to one of the following formats: delta, iceberg, parquet, or orc. Example:

    CREATE DATABASE IF NOT EXISTS db_storage_format 
LOCATION 'oss://path/to/db/' 
WITH DBPROPERTIES ('storage.format'='delta');

    After you execute this statement, tables created in the db_storage_format database default to the delta format. If you explicitly specify a table format by using using ${tableFormat} when creating a table, the explicitly specified format takes precedence.

Data lake table operations

Based on your chosen engine and lake format, refer to the corresponding documentation to create tables and perform read/write operations:

Previous：Open data lakeNext：Create data lake tables