Supported real-time data sources
In the Dataphin real-time compute engine, you can use real-time tasks to create table types that correspond to real-time data sources. This topic describes the real-time data sources supported by the engine and their read/write support.
Real-time data sources for Ververica Flink
|
Storage type |
Data source type |
Meta table |
Read |
Write |
Dimension table read |
|
Big data storage |
MaxCompute |
Supported |
Supported |
Supported |
Supported |
|
Hologres |
Supported |
Supported |
Supported |
- |
|
|
Hudi |
Supported |
Supported |
Supported |
- |
|
|
StarRocks |
Supported |
Supported |
Supported |
- |
|
|
Lindorm (Compute Engine) |
- |
- |
Supported |
Supported |
|
|
Relational database |
PolarDB |
- |
Supported |
Supported |
- |
|
PolarDB-X (formerly DRDS) |
Supported |
- |
Supported |
Supported |
|
|
MySQL |
Supported |
Supported (CDC) |
Supported |
Supported |
|
|
PostgreSQL |
Supported |
Supported |
- |
- |
|
|
AnalyticDB for MySQL 3.0 |
Supported |
- |
Supported |
Supported |
|
|
AnalyticDB for PostgreSQL |
Supported |
- |
Supported |
Supported |
|
|
Oracle |
- |
- |
Supported |
- |
|
|
ClickHouse |
Supported |
- |
Supported |
- |
|
|
Message queue |
Log Service |
Supported |
Supported |
Supported |
- |
|
Kafka |
Supported |
Supported |
Supported |
- |
|
|
RocketMQ |
Supported |
Supported |
Supported |
Supported |
|
|
NoSQL |
Elasticsearch |
Supported |
Supported |
Supported |
Supported |
|
Alibaba Cloud HBase |
Supported |
- |
Supported |
Supported |
|
|
Tablestore |
Supported |
- |
Supported |
Supported |
|
|
Redis |
Supported |
- |
Supported |
Supported |
|
|
Lindorm (Wide Table) |
Supported |
- |
Supported |
Supported |
Supported data sources for Alibaba Blink
|
Storage type |
Data source type |
Read |
Write |
Dimension table read |
|
Big data storage |
MaxCompute |
Supported |
Supported |
Supported |
|
DataHub |
Supported |
Supported |
- |
|
|
Hologres |
Supported |
Supported |
Supported |
|
|
Relational database |
PolarDB-X (formerly DRDS) |
- |
Supported |
Supported |
|
AnalyticDB for MySQL 2.0 |
- |
Supported |
- |
|
|
AnalyticDB for MySQL 3.0 |
- |
Supported |
Supported |
|
|
Message queue |
Log Service (SLS/LOG_HUB) |
Supported |
Supported |
- |
|
Kafka |
Supported |
Supported |
- |
|
|
RocketMQ |
Supported |
Supported |
- |
|
|
NoSQL |
Elasticsearch |
Supported |
Supported |
- |
|
Alibaba Cloud HBase |
- |
Supported |
Supported |
|
|
Tablestore |
- |
Supported |
Supported |
|
|
Redis |
- |
Supported |
Supported |
Supported real-time data sources
|
Storage type |
Data source type |
Meta table |
Read |
Write |
Dimension table read |
|
Big data storage |
Hive |
Supported (only for Hive 2.1.1 on CDH 6.x, Hive 3.1.3 on CDP 7.x, and Hive 3.1.0 on AsiaInfo DP 5.x) |
Supported |
Supported |
Supported |
|
Hive (Hudi) |
Supported |
Supported |
Supported |
- |
|
|
Hive (Iceberg) |
- |
Supported (native DDL only) |
Supported (native DDL only) |
- |
|
|
Hive (Paimon) |
Supported (only for Hive 3.1.x on E-MapReduce 5.x, Hive 2.1.1 on CDH 6.x, and Amazon EMR) |
Supported. Requirements depend on the platform: E-MapReduce 5.x (Hive 3.1.x): Apache Flink 1.20.1, Paimon 1.0.1; CDH 6.x (Hive 2.1.1): Apache Flink 1.15.3, Paimon 0.8.2; Amazon EMR: Apache Flink 1.20, Paimon 1.3.1. |
Supported. Requirements depend on the platform: E-MapReduce 5.x (Hive 3.1.x): Apache Flink 1.20.1, Paimon 1.0.1; CDH 6.x (Hive 2.1.1): Apache Flink 1.15.3, Paimon 0.8.2; Amazon EMR: Apache Flink 1.20, Paimon 1.3.1. |
Supported. Requirements depend on the platform: E-MapReduce 5.x (Hive 3.1.x): Apache Flink 1.20.1, Paimon 1.0.1; CDH 6.x (Hive 2.1.1): Apache Flink 1.15.3, Paimon 0.8.2; Amazon EMR: Apache Flink 1.20, Paimon 1.3.1. |
|
|
Hudi |
Supported |
Supported |
Supported |
- |
|
|
Doris |
Supported |
Supported (with Apache Flink 1.14 and 1.15) |
Supported (with Apache Flink 1.14 and 1.15) |
Supported |
|
|
SelectDB |
- |
- |
Supported |
- |
|
|
Paimon |
Supported |
Supported (with Apache Flink 1.14 and 1.15) |
Supported (with Apache Flink 1.14 and 1.15) |
- |
|
|
StarRocks |
Supported |
Supported |
Supported |
- |
|
|
TDH Inceptor (9.3.x) |
Supported |
- |
Supported |
Supported |
|
|
Relational data source |
MySQL |
Supported |
Supported (CDC) |
Supported |
Supported |
|
SAP HANA |
Supported |
Supported |
- |
- |
|
|
Microsoft SQL Server |
Supported |
Supported (CDC) |
Supported (only with Apache Flink 1.20.x) |
- |
|
|
PostgreSQL |
Supported |
Supported |
Supported |
Supported |
|
|
AnalyticDB for PostgreSQL |
Supported |
- |
Supported |
- |
|
|
OceanBase |
Supported |
Supported (CDC) |
- |
- |
|
|
Oracle |
Supported |
Supported (CDC) |
Supported |
- |
|
|
ClickHouse |
Supported |
- |
Supported (only with Apache Flink 1.15) |
- |
|
|
TiDB |
Supported |
Supported (CDC) |
- |
- |
|
|
GaussDB (DWS) |
- |
- |
Supported |
Supported |
|
|
TDSQL for PostgreSQL |
Supported (only with Apache Flink 1.20) |
Supported (CDC, only with Apache Flink 1.20) |
Supported (only with Apache Flink 1.20) |
- |
|
|
Message queue |
Log Service |
Supported |
Supported (only with Apache Flink 1.15) |
Supported (only with Apache Flink 1.15) |
- |
|
Kafka |
Supported |
Supported |
Supported |
- |
|
|
RabbitMQ |
Supported |
Supported |
Supported |
- |
|
|
RocketMQ |
Supported |
Supported |
Supported |
- |
|
|
NoSQL |
HBase 0.9.4 |
- |
- |
Supported |
Supported |
|
HBase 1.1.x |
Supported |
- |
Supported |
Supported |
|
|
HBase 2.x |
Supported |
- |
Supported |
Supported |
|
|
Elasticsearch |
Supported |
- |
Supported |
- |
|
|
Redis |
Supported |
- |
Supported (only with Apache Flink 1.15) |
Supported (only with Apache Flink 1.15) |
|
|
MongoDB |
Supported |
Supported (CDC) |
- |
- |
|
|
Semi-structured storage |
SAP TABLE |
- |
Supported |
- |
- |
|
File |
OSS |
Supported |
- |
Supported |
- |
Real-time data sources for FusionInsight Flink
|
Storage type |
Data source type |
Read |
Write |
Dimension table read |
|
Relational database |
MySQL |
Supported (CDC) |
Supported (JDBC) |
Supported (JDBC) |
|
PostgreSQL |
- |
Supported (JDBC) |
Supported (JDBC) |
|
|
Oracle |
- |
Supported (JDBC) |
Supported (JDBC) |
|
|
Message queue |
Kafka |
Supported |
Supported |
- |