Data management overview

Updated at:

DLF data management has two categories: Paimon Catalog and Lance Catalog. Choose a catalog based on the tables you need to manage, then find the corresponding document.

Paimon Catalog

Paimon Catalog manages Paimon, Iceberg, Format, and Object tables. This section links to documentation for databases, views, functions, tables, the recycle bin, and data sharing.

Select a document by task

Choose a document based on the data object you want to manage or the task you want to perform.

Goal

Document

Create and manage databases

Databases

Select a table type based on your data and use case

Select a table type

Manage views or functions

Views and Functions

Restore deleted databases, tables, or files

Recycle bin

Share data with other accounts

Data sharing

Select a table type

Paimon Catalog manages the following table types. Select a document based on your data type and use case.

Table type

Use case

Document

Paimon Table

Manage Paimon tables.

Paimon Table

Iceberg Table

Manage Iceberg tables.

Iceberg tables

Format Table

Manage Hive-compatible tables that use formats such as Parquet, CSV, and ORC.

Format tables

Object Table

Manage non-tabular datasets such as images and videos.

Object Table

Data object hierarchy

Paimon Catalog organizes metadata resources into the following hierarchy:

  1. Catalog: The top-level logical metadata entity. You can use catalogs to isolate metadata for different businesses or users and to manage permissions and data lake resources.

  2. Database: A logical metadata group in a catalog. Databases provide finer-grained data organization and access control.

  3. Table: DLF supports multiple table types to centrally manage data in different formats.

  4. View: A view persisted in DLF. You can configure SQL dialects for different compute engines.

  5. Function: A function persisted in DLF.

Metadata service

Paimon Catalog centrally manages table, view, and function metadata. Alibaba Cloud big data and AI engines can use this catalog to access that metadata.

  • Data lake formats: DLF supports Apache Paimon, Apache Iceberg, and their ecosystem components.

  • Unstructured data: Object Table lets you manage non-tabular datasets such as images and videos.

  • Standard file formats: DLF supports Hive-compatible tables that use formats such as Parquet, CSV, and ORC.

DLF also provides table management APIs that comply with the Apache Paimon REST Catalog standard. Its file storage structure is compatible with the open source community. Paimon-compatible engines and applications can create, update, query, and delete Paimon tables in DLF.

Lance Catalog

DLF Lance Catalog provides a fully managed catalog and storage for AI and multimodal applications. It manages its own namespaces and tables. Read the following documents to learn how it works and how to create, connect to, and manage a Lance Catalog.