Data management overview
DLF data management has two categories: Paimon Catalog and Lance Catalog. Choose a catalog based on the tables you need to manage, then find the corresponding document.
Paimon Catalog
Paimon Catalog manages Paimon, Iceberg, Format, and Object tables. This section links to documentation for databases, views, functions, tables, the recycle bin, and data sharing.
Select a document by task
Choose a document based on the data object you want to manage or the task you want to perform.
Goal | Document |
Create and manage databases | |
Select a table type based on your data and use case | |
Manage views or functions | |
Restore deleted databases, tables, or files | |
Share data with other accounts |
Select a table type
Paimon Catalog manages the following table types. Select a document based on your data type and use case.
Table type | Use case | Document |
Paimon Table | Manage Paimon tables. | |
Iceberg Table | Manage Iceberg tables. | |
Format Table | Manage Hive-compatible tables that use formats such as Parquet, CSV, and ORC. | |
Object Table | Manage non-tabular datasets such as images and videos. |
Data object hierarchy
Paimon Catalog organizes metadata resources into the following hierarchy:
Catalog: The top-level logical metadata entity. You can use catalogs to isolate metadata for different businesses or users and to manage permissions and data lake resources.
Database: A logical metadata group in a catalog. Databases provide finer-grained data organization and access control.
Table: DLF supports multiple table types to centrally manage data in different formats.
View: A view persisted in DLF. You can configure SQL dialects for different compute engines.
Function: A function persisted in DLF.
Metadata service
Paimon Catalog centrally manages table, view, and function metadata. Alibaba Cloud big data and AI engines can use this catalog to access that metadata.
Data lake formats: DLF supports Apache Paimon, Apache Iceberg, and their ecosystem components.
Unstructured data: Object Table lets you manage non-tabular datasets such as images and videos.
Standard file formats: DLF supports Hive-compatible tables that use formats such as Parquet, CSV, and ORC.
DLF also provides table management APIs that comply with the Apache Paimon REST Catalog standard. Its file storage structure is compatible with the open source community. Paimon-compatible engines and applications can create, update, query, and delete Paimon tables in DLF.
Lance Catalog
DLF Lance Catalog provides a fully managed catalog and storage for AI and multimodal applications. It manages its own namespaces and tables. Read the following documents to learn how it works and how to create, connect to, and manage a Lance Catalog.
DLF Lance overview and how it works: Learn about Lance Catalog capabilities and how it works.
Manage and use a Lance Catalog: Learn how to create, connect to, and manage a Lance Catalog.