Paimon data source

Prerequisites

Before you begin, make sure that:

• All Paimon cluster nodes and the SelectDB instance are in the same virtual private cloud (VPC). If they reside in different VPCs, resolve network connectivity first. For details, see What do I do if a connection fails to be established between an ApsaraDB for SelectDB instance and a data source?

• The IP addresses of all Paimon cluster nodes are added to the IP address whitelist of the SelectDB instance. For details, see Configure an IP address whitelist.

• The VPC IP addresses of the SelectDB instance are added to the Paimon cluster's IP address whitelist (if the Paimon cluster supports a whitelist mechanism). To find the VPC IP addresses of your SelectDB instance, see How do I view the IP addresses in the VPC to which my ApsaraDB SelectDB instance belongs?

• You understand how catalogs work in SelectDB. For background, see Data lakehouse.

Usage notes

• Supported Paimon versions depend on your SelectDB version:

  • SelectDB V3.0: Paimon 0.7 and earlier

  • SelectDB V4.0: Paimon 0.8 and earlier

• OSS-backed Paimon warehouses are supported in ApsaraDB for SelectDB V3.0.6 and later.

Step 1: Connect to a SelectDB instance

Connect to your SelectDB instance using a MySQL client. For details, see Connect to an ApsaraDB for SelectDB instance by using a MySQL client.

Step 2: Create a Paimon catalog

SelectDB integrates external data sources through external catalogs. Paimon supports two metastore types:

• File system metastore (default): stores both Paimon metadata and data in a file system.

• Hive metastore: stores Paimon metadata as tables in a Hive metastore, queryable directly from Hive.

Create a catalog based on your metastore type and storage backend.

CREATE CATALOG `paimon_catalog` PROPERTIES (
    "type" = "paimon",
    "warehouse" = "hdfs://HDFS8000871/user/paimon",
    "dfs.nameservices" = "HDFS8000871",
    "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
    "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
    "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
    "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
    "hadoop.username" = "hadoop"
);

CREATE CATALOG `paimon_oss_catalog` PROPERTIES (
    "type" = "paimon",
    "warehouse" = "oss://paimon-zd/paimonoss",
    "oss.endpoint" = "oss-cn-beijing.aliyuncs.com",
    "oss.access_key" = "<your-access-key-id>",
    "oss.secret_key" = "<your-access-key-secret>"
);

CREATE CATALOG `paimon_hms` PROPERTIES (
    "type" = "paimon",
    "paimon.catalog.type" = "hms",
    "warehouse" = "hdfs://HDFS8000871/user/zhangdong/paimon2",
    "hive.metastore.uris" = "thrift://172.21.0.44:7004",
    "dfs.nameservices" = "HDFS8000871",
    "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
    "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
    "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
    "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
    "hadoop.username" = "hadoop"
);

Step 3: Verify the catalog

Run SHOW CATALOGS to confirm the catalog was created:

SHOW CATALOGS;

Expected output:

+--------------+----------------+----------+-----------+-------------------------+---------------------+------------------------+
| CatalogId    | CatalogName    | Type     | IsCurrent | CreateTime              | LastUpdateTime      | Comment                |
+--------------+----------------+----------+-----------+-------------------------+---------------------+------------------------+
| 436009309195 | paimon_catalog | paimon   |           | 2024-08-06 17:09:08.058 | 2024-07-19 18:04:37 |                        |
|            0 | internal       | internal | yes       | UNRECORDED              | NULL                | Doris internal catalog |
+--------------+----------------+----------+-----------+-------------------------+---------------------+------------------------+

Step 4: Query Paimon data

After creating the catalog, query Paimon data using any of the following approaches:

-- Option 1: Switch to the catalog, then use a database
SWITCH paimon_catalog;
USE test_db;
SELECT * FROM test_t;

-- Option 2: Use a fully qualified table name without switching
SELECT * FROM paimon_catalog.test_db.test_t;

To explore available databases and tables:

SHOW DATABASES;
SHOW TABLES;

Column data type mappings