Dataphin data sources-Dataphin(Dataphin)-阿里云帮助中心

Before using Dataphin, select a database or data warehouse to use as a data source for reading raw data and writing output during data development. Dataphin integrates with various data engines, including data warehouses such as MaxCompute and Hive, and traditional enterprise databases like MySQL and Oracle.

Background

Dataphin connects to various data sources, such as big data storage, file, message queue, relational, and NoSQL data sources.

To connect a data source in Dataphin, you must first create it in data source management.
Dataphin lets you add both production data sources and development data sources. Basic projects and the Prod environment of Dev-Prod projects read from and write to production data sources. The Dev environment of Dev-Prod projects reads from and writes to development data sources. For data service, the Prod environment of both Basic and Dev-Prod projects reads from production data sources, while the Dev environment of Dev-Prod projects reads from development data sources. Synchronization tasks do not support a dual production and development environment and can only read from and write to production data sources.
Note
If the data source type you need is not built-in, you can create a custom offline or real-time source type to connect your data source to Dataphin. For details, see:
- Create an offline custom source type
- Create a real-time custom source type

Data sources

Use case	Description	Related documents
Offline integration	Offline integration supports various components, including input, output, and transformation components. You can create a single offline integration pipeline by dragging, configuring, and assembling these components on a canvas. Offline integration also supports a script mode for more personalized configuration. Additionally, when you create a custom RDBMS data source, its input and output components are automatically added to the component library to meet diverse data synchronization needs.	Offline integration
Offline development - Database SQL	After you connect a data source to Dataphin, you can create database SQL tasks in Dataphin for development.	Create a database SQL task
Metadata collection	The Metadata Center extracts, processes, and centrally manages metadata from various business systems. This supports data governance and helps your organization organize, retrieve, and analyze data.	Metadata Center
Real-time development	You can use connected data sources to create real-time metatables and develop real-time tasks.	Flink SQL task development
Data quality	Asset Quality is a comprehensive data quality solution in Dataphin that provides end-to-end support for data development and usage. You can create global table quality rules or data source-specific quality rules. Data source-specific quality rules allow you to create monitoring rules for any data source within Dataphin. You can run a connectivity test for all these data sources. However, only some data sources support quality rules for monitoring table schema changes. For details, see the Data source quality - table schema change column in the table below.	Data quality overview
OneService	OneService is the final step in building a data middle platform with Dataphin. It acts as a unified service portal, providing centralized, marketplace-style data management. This simplifies data access while ensuring data security.	OneService overview
Tag Factory	Tag Factory provides a one-stop platform for the entire tag lifecycle, from creation to service delivery. It allows data R&D teams and developers to develop, manage, explore, and serve offline, real-time, and service-based tags for various scenarios, such as risk control and marketing. Tag Factory empowers upstream business applications, helps enterprises build their tag assets, and makes tags efficient to develop, easy to find, simple to use, and convenient to manage.	Tag Factory overview

This topic lists only the data sources that can be connected to Dataphin and their supported use cases. For details about the specific features supported by each data source in different scenarios, see the following documents:

Big data storage sources

Type	Batch integration	Real-time integration	Offline SQL	Metadata collection	Real-time development	Global table quality	Schema change quality	Data service	Tag factory	Guide
MaxCompute	Supported	Supported	Not supported	Not supported	Supported	Supported	Supported	Supported	Supported	Create a MaxCompute data source
Hive	Supported	Supported	Not supported	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create a Hive data source
Hologres	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Create a Hologres data source
Impala	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Supported	Supported	Not supported	Create an Impala data source
StarRocks	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Not supported	Create a StarRocks data source
Hudi	Not supported	Not supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Not supported	Create a Hudi data source
Doris	Supported	Not supported	Supported	Supported	Supported	Supported	Supported	Supported	Not supported	Create a Doris data source
Greenplum	Supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Not supported	Supported	Create a Greenplum data source
SelectDB	Supported	Not supported	Supported	Supported	Not supported	Supported	Supported	Supported	Not supported	Create a SelectDB data source
Databricks	Supported	Supported	Not supported	Not supported	Not supported	Supported	Supported	Supported	Not supported	Create a Databricks data source
Amazon Redshift	Supported	Not supported	Not supported	Supported	Not supported	Supported	Supported	Supported	Not supported	Create an Amazon Redshift data source
DolphinDB	Supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Supported	Not supported	Create a DolphinDB data source
Snowflake	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create a Snowflake data source
Data Lake Formation	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create a Data Lake Formation data source

File data sources

Data source type	Offline integration	Real-time integration	Offline SQL development	Metadata collection	Real-time development	Global table quality	Schema changes	Data service	Tag factory	Creation guide
FTP	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create an FTP data source
OSS	Supported	Not supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Not supported	Create an OSS data source
Amazon S3	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create an Amazon S3 data source

Message queue data sources

Data source type	Batch integration	Real-time integration	Batch SQL	Metadata collection	Real-time development	Global table quality	Schema volatility	Data service	Tag factory	Actions
Log Service	Supported	Not supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Not supported	Create Log Service data source
Kafka	Supported	Supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Supported	Create Kafka data source
RocketMQ	Not supported	Not supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Not supported	Create RocketMQ data source
DataHub	Supported	Supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Supported	Create DataHub data source
RabbitMQ	Not supported	Not supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Not supported	Create RabbitMQ data source

Relational data source

Data source type	Batch integration	Real-time integration	Offline SQL	Metadata collection	Real-time development	Global table quality	Schema change	Data service	Label factory	Unstructured dataset	Guide
PolarDB	Supported	Not supported	Not supported	Not supported	Supported	Supported	Supported	Not supported	Not supported	Not supported	Create a PolarDB data source
PolarDB-X	Supported	Not supported	Not supported	Supported	Supported	Supported	Supported	Not supported	Not supported	Not supported	Create a PolarDB-X data source
PolarDB-X 2.0	Supported	Not supported	Supported	Not supported	Not supported	Supported	Supported	Supported	Not supported	Not supported	Create a PolarDB-X 2.0 data source
MySQL	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Not supported	Create a MySQL data source
SAP HANA	Supported	Not supported	Not supported	Supported	Supported	Supported	Supported	Supported	Not supported	Not supported	Create an SAP HANA data source
Microsoft SQL Server	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Not supported	Not supported	Create a Microsoft SQL Server data source
PostgreSQL	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Create a PostgreSQL data source
AnalyticDB for MySQL 2.0	Supported	Not supported	Supported	Not supported	Supported	Not supported	Supported	Supported	Not supported	Not supported	Create an AnalyticDB for MySQL 2.0 data source
AnalyticDB for MySQL 3.0	Supported	Not supported	Supported	Supported	Supported	Not supported	Supported	Supported	Not supported	Not supported	Create an AnalyticDB for MySQL 3.0 data source
AnalyticDB for PostgreSQL	Supported	Not supported	Supported	Not supported	Supported	Supported	Supported	Supported	Supported	Not supported	Create an AnalyticDB for PostgreSQL data source
OceanBase	Supported	Not supported	Supported	Supported	Supported	Not supported	Supported	Supported	Not supported	Not supported	Create an OceanBase data source
Oracle	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Supported	Not supported	Create an Oracle data source
Vertica	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Create a Vertica data source
IBM DB2	Supported	Supported	Not supported	Supported	Not supported	Supported	Supported	Not supported	Not supported	Not supported	Create an IBM DB2 data source
Teradata	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Create a Teradata data source
ClickHouse	Supported	Not supported	Supported	Supported	Supported	Supported	Supported	Supported	Not supported	Not supported	Create a ClickHouse data source
DM	Supported	Not supported	Supported	Supported	Not supported	Supported	Supported	Supported	Not supported	Not supported	Create a DM data source
GBase 8a	Supported	Not supported	Supported	Not supported	Not supported	Not supported	Not supported	Supported	Not supported	Not supported	Create a GBase 8a data source
KingbaseES	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create a KingbaseES data source
TiDB	Supported	Not supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create a TiDB data source
GoldenDB	Supported	Not supported	Not supported	Not supported	Not supported	Supported	Supported	Not supported	Not supported	Not supported	Create a GoldenDB data source
openGauss	Supported	Not supported	Supported	Supported	Not supported	Not supported	Not supported	Not supported	Supported	Not supported	Create an openGauss data source
GaussDB(DWS)	Supported	Not supported	Supported	Not supported	Supported	Supported	Supported	Supported	Not supported	Not supported	Create a GaussDB(DWS) data source
Amazon RDS for MySQL	Supported	Not supported	Not supported	Supported	Not supported	Supported	Supported	Not supported	Not supported	Not supported	Create an Amazon RDS for MySQL data source
Amazon RDS for PostgreSQL	Supported	Not supported	Not supported	Supported	Not supported	Supported	Supported	Not supported	Not supported	Not supported	Create an Amazon RDS for PostgreSQL data source
Amazon RDS for SQL Server	Supported	Not supported	Not supported	Supported	Not supported	Supported	Supported	Not supported	Not supported	Not supported	Create an Amazon RDS for SQL Server data source
Amazon RDS for Oracle	Supported	Not supported	Not supported	Supported	Not supported	Supported	Supported	Not supported	Not supported	Not supported	Create an Amazon RDS for Oracle data source
Amazon RDS for DB2	Supported	Not supported	Not supported	Supported	Not supported	Supported	Supported	Not supported	Not supported	Not supported	Create an Amazon RDS for DB2 data source
TDSQL for MySQL	Supported	Not supported	Supported	Supported	Not supported	Supported	Supported	Supported	Not supported	Not supported	Create a TDSQL for MySQL data source
GBase 8c	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Supported	Not supported	Not supported	Create a GBase 8c data source
TDSQL for PostgreSQL	Supported	Not supported	Supported	Not supported	Not supported	Supported	Not supported	Supported	Not supported	Not supported	Create a TDSQL for PostgreSQL data source

NoSQL data source

Data source type	Batch integration	Real-time integration	Batch development (SQL)	Metadata collection	Real-time development	Global table quality	Schema changes	Data service	Tag factory	Knowledge graph	Actions
Elasticsearch	Supported	Not supported	Not supported	Supported	Supported	Not supported	Not supported	Supported	Not supported	Not supported	Create Elasticsearch data source
MongoDB	Supported	Not supported	Not supported	Not supported	Supported	Not supported	Not supported	Supported	Not supported	Not supported	Create MongoDB data source
Tablestore	Supported	Not supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Supported	Not supported	Create Tablestore data source
ApsaraDB for HBase	Not supported	Not supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create ApsaraDB for HBase data source
Redis	Supported	Not supported	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create Redis data source
Presto	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create Presto data source
Easysearch	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create Easysearch data source
Trino	Not supported	Not supported	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create Trino data source
OpenSearch	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create OpenSearch data source
InfluxDB	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create InfluxDB data source
Lindorm	Supported	Not supported	Supported (excludes wide-column engine-HBase connections)	Not supported	Not supported	Not supported	Not supported	Supported	Supported (wide-column engine-HBase connections only)	Supported	Create Lindorm data source

Semi-structured data sources

Data source type	Offline integration	Real-time integration	Offline SQL	Metadata collection	Real-time development	Global table quality	Schema change	Data service	Label factory	Knowledge graph	Guide
API	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Supported	Not supported	Create API Data Source
SAP Table	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create SAP Table Data Source
Feishu Bitable data source	Supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Create a Feishu Bitable data source
Neo4j	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Not supported	Supported	Create Neo4j Data Source

Vector database

Data source type

Offline integration

Real-time integration

Offline SQL development

Metadata collection

Real-time development

Global table quality

Table schema change

Data service

Tag factory

Knowledge graph

Unstructured dataset

Creation guide

Milvus

Supported

Not supported

Supported