Paimon Catalog
The Paimon Catalog data source collects and governs metadata for Paimon tables that do not originate from Data Lake Formation (DLF), enabling unified governance of Paimon data lake assets in Data Map.
Introduction
As the lakehouse architecture gains traction, open table formats such as Paimon, Iceberg, and Delta Lake have become foundational for building real-time data warehouses and enabling unified batch-stream processing. Paimon Catalog is widely used in the Flink stream processing ecosystem due to its native compatibility.
DataWorks integrates with Data Lake Formation for unified management of data lake tables through DLF data sources. However, a user might define a Paimon Catalog using the Flink engine with metadata and data stored in Alibaba Cloud OSS, outside DLF management.
The Paimon Catalog data source bridges this gap by enabling metadata collection and governance for native, non-DLF-managed lake formats. It makes self-declared catalog data visible, manageable, and usable across the lakehouse.
Limitations
-
Network connectivity: Only a serverless resource group is supported.
-
Scenarios: Paimon Catalog is currently used only for Collect Metadata and governance. It does not support data integration and synchronization tasks. To read from and write to Paimon tables for data synchronization, use other data sources, such as DLF or OSS.
Procedure
1. Go to the Data Sources page
-
Log on to the DataWorks console and switch to the target region. In the left navigation bar, click Workspace, and then click Manage in the Actions column of the target workspace to go to the management page.
-
On the workspace management center page, click Data Sources in the left navigation bar to go to the Data Source page.
2. Add a Paimon Catalog data source
-
On the Data Sources page, click Add Data Source .
-
In the Add Data Source dialog box, search for and select Paimon Catalog.
3. Configure parameters
Configure the following parameters:
|
Parameter |
Description |
|
Data Source Name |
Specify a custom data source name, such as |
|
Catalog |
The catalog name for the connection, such as |
|
MetaStore |
The storage type of the catalog. Currently, only Filesystem is supported. |
|
Filesystem |
The file storage type. Currently, only OSS is supported. |
|
Access Mode |
|
|
Region |
Select a bucket in the same region as the workspace for optimal performance. For cross-region data sources, establish a VPC peering connection. For details, see Connect to a data source in a different region under the same Alibaba Cloud account. Alternatively, connect by using a public endpoint. |
|
Endpoint |
For endpoint details, see Access domain names and data centers. |
|
Warehouse |
Warehouse path: The storage path of the Paimon Catalog in OSS.
|
to the right of the input box to select a path from a list.4. Test connectivity
After configuration, run a connectivity test to verify the connection between the data source and the resource group.
-
If Connected is displayed, the configuration is correct.
-
If Connection failed. is displayed, a diagnostic tool opens to help you troubleshoot. Common causes include incorrect credentials, network connectivity issues such as an unconfigured IP address whitelist, or a missing NAT gateway.
-
In standard mode, you must ensure that both the development environment and the production environment are Connected. Otherwise, errors will occur during subsequent operations such as metadata collection.
Next steps
After adding the data source, go to Data Map to collect metadata. You can then view and govern the collected assets.