Data management overview-Data Lake Formation(DLF)-阿里云帮助中心

DLF organizes and centrally manages data lake metadata by using catalogs, databases, tables, views, and functions. Select a document based on the data object that you want to manage or the task that you want to perform.

Select a document by task

If you know the data object that you want to manage or the task that you want to perform, go directly to the corresponding document.

Goal	Document
Create and manage catalogs	Manage catalogs
Create and manage databases	Databases
Select a table type based on your data and use case	Select a table type
Manage views or functions	Views and Functions
Restore deleted databases, tables, or files	Recycle bin
Share data with other accounts	Data sharing

Select a table type

DLF supports multiple data lake table formats and file formats. Select a document based on your data type and use case.

Table type	Use case	Document
Paimon Table	Manage Paimon tables.	Paimon Table
Iceberg Table	Manage Iceberg tables.	Iceberg tables
Format Table	Manage Hive-compatible tables that use formats such as Parquet, CSV, and ORC.	Format tables
Lance Table	Manage data in the Lance format for high-performance AI retrieval and training.	Lance Table
Object Table	Manage non-tabular datasets such as images and videos.	Object Table

Data object hierarchy

DLF organizes metadata resources into the following hierarchy:

Catalog: The top-level logical metadata entity. You can use catalogs to isolate metadata for different businesses or users and to manage permissions and data lake resources.
Database: A logical metadata group in a catalog. Databases provide finer-grained data organization and access control.
Table: DLF supports multiple table types to centrally manage data in different formats.
View: A view persisted in DLF. You can configure SQL dialects for different compute engines.
Function: A function persisted in DLF.

Unified metadata service

DLF provides a unified metadata management service. Alibaba Cloud big data and AI engines can use the same catalog system to access the metadata of tables, views, and functions managed by DLF.

Data lake formats: DLF supports Apache Paimon, Apache Iceberg, and their ecosystem components.
AI and vector data: DLF supports the Lance format for high-performance AI retrieval and training.
Unstructured data: Object Table lets you manage non-tabular datasets such as images and videos.
Standard file formats: DLF supports Hive-compatible tables that use formats such as Parquet, CSV, and ORC.

DLF also provides table management APIs that comply with the Apache Paimon REST Catalog standard. Its file storage structure is compatible with the open source community. Paimon-compatible engines and applications can create, update, query, and delete Paimon tables in DLF.