文档

Paimon数据源

更新时间:
一键部署

本文介绍云数据库 SelectDB 版与Paimon数据源进行对接使用的流程,指导您对Paimon数据源进行联邦分析。

注意事项

当前适配的Paimon版本为0.5.0。

创建Catalog

Paimon Catalog支持基于两种Metastore类型创建Catalog:

  • Filesystem(默认):元数据和数据均存储在Filesystem。

  • Hive Metastore:元数据存储在Hive Metastore中,您可以直接从Hive访问这些表。

基于FileSystem创建Catalog

说明

SelectDB 2.X及之前版本,请参见基于Hive Metastore创建Catalog

HDFS

CREATE CATALOG `paimon_hdfs` PROPERTIES (
    "type" = "paimon",
    "warehouse" = "hdfs://HDFS8000871/user/paimon",
    "dfs.nameservices" = "HDFS8000871",
    "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
    "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
    "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
    "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
    "hadoop.username" = "hadoop"
);

OSS

云数据库 SelectDB 版 3.0.6版本开始,支持以OSS作为数据来源进行Paimon对接。

CREATE CATALOG `paimon_oss` PROPERTIES (
    "type" = "paimon",
    "warehouse" = "oss://paimon-zd/paimonoss",
    "oss.endpoint" = "oss-cn-beijing.aliyuncs.com",
    "oss.access_key" = "ak",
    "oss.secret_key" = "sk"
);

基于Hive Metastore创建Catalog

CREATE CATALOG `paimon_hms` PROPERTIES (
    "type" = "paimon",
    "paimon.catalog.type" = "hms",
    "warehouse" = "hdfs://HDFS8000871/user/zhangdong/paimon2",
    "hive.metastore.uris" = "thrift://172.21.0.44:7004",
    "dfs.nameservices" = "HDFS8000871",
    "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
    "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
    "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
    "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
    "hadoop.username" = "hadoop"
);

列类型映射

Paimon Data Type

SelectDB Data Type

备注

BooleanType

Boolean

TinyIntType

TinyInt

SmallIntType

SmallInt

IntType

Int

FloatType

Float

BigIntType

BigInt

DoubleType

Double

VarCharType

VarChar

CharType

Char

DecimalType(precision, scale)

Decimal(precision, scale)

TimestampType,LocalZonedTimestampType

DateTime

DateType

Date

MapType

Map

支持Map嵌套

ArrayType

Array

支持Array嵌套

VarBinaryType, BinaryType

Binary

  • 本页导读 (1)