Dataphin data sources
Before you can use Dataphin, you must select a database or data warehouse that meets your business requirements to serve as a data source for reading raw data and writing output data. Dataphin integrates a variety of data engines and supports data warehouses such as MaxCompute and Hive, as well as traditional enterprise databases like MySQL and Oracle.
Background
Dataphin supports various data sources, including big data storage sources, file data sources, message queue data sources, relational data sources, and NoSQL data sources. The data source types supported by each module are as follows:
To connect to a data source in Dataphin, you must first create it in data source management.
Dataphin allows you to add both production and development data sources. Basic projects and the Prod environment of Dev-Prod projects read from and write to production data sources. The Dev environment of Dev-Prod projects reads from and writes to development data sources. In DataService, Basic projects and the Prod environment of Dev-Prod projects read from production data sources, while the Dev environment of Dev-Prod projects reads from development data sources. Synchronization tasks do not support separate Prod and Dev environments and always read from and write to production data sources.
NoteIf a required data source type is not available as a built-in option, you can define a custom offline or real-time data source type and connect it to Dataphin. For detailed instructions, see the following topics:
Data sources
Use case | Description | References |
Batch integration | You can create a batch integration pipeline on a canvas by dragging, configuring, and connecting these components. It also supports a script mode for custom configurations. When you create a custom RDBMS data source, its input and output components are automatically added to the component library for data synchronization. | |
Real-time integration | Dataphin supports real-time integration, allowing you to synchronize data changes from an entire source database or specific tables with a destination data source. | |
Batch development - database SQL | After connecting a data source, you can create database SQL tasks in Dataphin. | |
Metadata collection | The metadata center extracts, processes, centrally stores, and manages metadata from various business systems to support data governance, enhancing your ability to organize, retrieve, and analyze data. | |
Real-time development | You can use connected data sources to create real-time metatables and develop real-time tasks. | |
Data quality | Asset quality, a feature in Dataphin, offers a comprehensive data quality solution for data development and consumption. You can create quality rules for global tables or specific data sources. Data source quality rules allow you to create quality monitoring rules for any data source in Dataphin. All supported data sources can run a connectivity test, but only some support quality rules for schema change detection. For details, see the Data source quality - Schema change detection column in the table below. | |
OneService | OneService is the final step in building a data middle platform with Dataphin. As a unified exit point for data services, OneService enables centralized data management. It simplifies data sharing while ensuring security. | |
Label factory | Label factory provides an end-to-end workflow from label creation to label serving. It offers a one-stop platform for data development teams and engineers, suitable for various use cases such as risk control and marketing. It supports the development, management, exploration, and serving of batch, real-time, and service-based labels. This helps enterprises build a repository of label assets and makes labels easier to develop, find, use, and manage. |
This topic lists the data sources that can be connected to Dataphin and their supported use cases. For details on the features supported by each data source for each use case, see the following topics:
Big data storage
Data source type | Batch integration | Real-time integration | Offline development - SQL | Metadata collection | Real-time development | Global table quality | Table structure change | Data service | Tag factory | Guide |
MaxCompute | Supported | Supported | Not supported | Not supported | Supported | Supported | Supported | Supported | Supported | |
Hive | Supported | Supported | Not supported | Supported | Supported | Supported | Supported | Supported | Not supported | |
Hologres | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | |
Impala | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported | Supported | Not supported | |
TDH Inceptor | Supported | Not supported | Not supported | Not supported | Supported | Supported | Supported | Supported | Not supported | |
Kudu | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | |
StarRocks | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Not supported | |
Hudi | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | |
Doris | Supported | Not supported | Supported | Supported | Supported | Supported | Supported | Supported | Not supported | |
Greenplum | Supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | Supported | |
TDengine | Supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | |
ArangoDB | Supported | Not supported | Not supported | Not supported | Not supported | Supported | Supported | Not supported | Not supported | |
SelectDB | Supported | Not supported | Supported | Supported | Not supported | Supported | Supported | Supported | Not supported | |
Databricks | Supported | Supported | Not supported | Not supported | Not supported | Supported | Supported | Supported | Not supported | |
Amazon Redshift | Supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Supported | Not supported | |
DolphinDB | Supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Supported | Not supported | |
Snowflake | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
AWS Lake Formation | Supported | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported |
File data sources
Data source type | Offline integration | Real-time integration | Offline development - SQL | Metadata collection | Real-time development | Global table quality | Data source quality - schema change | Data service | Label Factory | Setup guide |
HDFS | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
FTP | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
OSS | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | |
Amazon S3 | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported |
Message queue data sources
Data source type | Batch integration | Streaming integration | Batch development - database SQL | Metadata collection | Streaming development | Global table quality | Data source quality - table schema change | Data service | Tag factory | Actions |
Log Service | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | |
Kafka | Supported | Supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Supported | |
DataHub | Supported | Supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Supported | |
RabbitMQ | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported |
Relational data source
Data source type | Batch integration | Real-time integration | Batch development - SQL | Metadata collection | Real-time development | Global table quality | Data source quality - schema change | Data service | Tag factory | Unstructured dataset | Actions |
PolarDB | Supported | Not supported | Not supported | Not supported | Supported | Supported | Supported | Not supported | Not supported | Not supported | |
PolarDB-X | Supported | Not supported | Not supported | Supported | Supported | Supported | Supported | Not supported | Not supported | Not supported | |
PolarDB-X 2.0 | Supported | Not supported | Supported | Not supported | Not supported | Supported | Supported | Supported | Not supported | Not supported | |
MySQL | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Not supported | |
SAP HANA | Supported | Not supported | Not supported | Supported | Supported | Supported | Supported | Supported | Not supported | Not supported | |
Microsoft SQL Server | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Not supported | Not supported | |
PostgreSQL | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | |
AnalyticDB for MySQL 2.0 | Supported | Not supported | Supported | Not supported | Supported | Not supported | Supported | Supported | Not supported | Not supported | |
AnalyticDB for MySQL 3.0 | Supported | Not supported | Supported | Supported | Supported | Not supported | Supported | Supported | Not supported | Not supported | |
AnalyticDB for PostgreSQL | Supported | Not supported | Supported | Not supported | Supported | Supported | Supported | Supported | Supported | Not supported | |
OceanBase | Supported | Not supported | Supported | Supported | Supported | Not supported | Supported | Supported | Not supported | Not supported | |
Oracle | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Supported | Not supported | |
Vertica | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | |
IBM DB2 | Supported | Supported | Not supported | Supported | Not supported | Supported | Supported | Not supported | Not supported | Not supported | |
Teradata | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | |
ClickHouse | Supported | Not supported | Supported | Supported | Supported | Supported | Supported | Supported | Not supported | Not supported | |
DM | Supported | Not supported | Supported | Supported | Not supported | Supported | Supported | Supported | Not supported | Not supported | |
GBase 8a | Supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | |
KingbaseES | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
TiDB | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
GoldenDB | Supported | Not supported | Not supported | Not supported | Not supported | Supported | Supported | Not supported | Not supported | Not supported | |
openGauss | Supported | Not supported | Supported | Supported | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | |
GaussDB (DWS) | Supported | Not supported | Supported | Not supported | Supported | Supported | Supported | Supported | Not supported | Not supported | |
Amazon RDS for MySQL | Supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Supported | Not supported | Not supported | |
Amazon RDS for PostgreSQL | Supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Supported | Not supported | Not supported | |
Amazon RDS for SQL Server | Supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Supported | Not supported | Not supported | |
Amazon RDS for Oracle | Supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Supported | Not supported | Not supported | |
Amazon RDS for DB2 | Supported | Not supported | Not supported | Supported | Not supported | Supported | Supported | Not supported | Not supported | Not supported | |
TDSQL for MySQL | Supported | Not supported | Supported | Supported | Not supported | Supported | Supported | Supported | Not supported | Not supported | |
GBase 8c | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | |
TDSQL for PostgreSQL | Supported | Not supported | Supported | Not supported | Not supported | Supported | Not supported | Supported | Not supported | Not supported | |
OushuDB | Supported | Supported | Supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported |
NoSQL data sources
Data source type | Offline integration | Real-time integration | Offline development - SQL | Metadata collection | Real-time development | Global table quality | Data source quality - schema change | Data service | Tag factory | Knowledge graph | Actions |
HBase 0.9.4 | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Supported | Supported | Not supported | |
HBase 1.1x | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Supported | Supported | Not supported | |
HBase 2.0 | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Supported | Supported | Not supported | |
Elasticsearch | Supported | Not supported | Not supported | Supported | Supported | Not supported | Not supported | Supported | supported | Not supported | |
MongoDB | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Supported | Not supported | Not supported | |
Tablestore | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Supported | Not supported | |
ApsaraDB for HBase | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
Redis | Supported | Not supported | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
Presto | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
Easysearch | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
Trino | Not supported | Not supported | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
OpenSearch | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
InfluxDB | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
Lindorm | Supported | Not supported | Supported (excluding the wide column engine - HBase connection type) | Not supported | Not supported | Not supported | Not supported | Supported | Supported (only for the wide column engine - HBase connection type) | Supported |
Semi-structured data sources
Data source type | Batch integration | Real-time integration | Batch development-SQL | Metadata collection | Real-time development | Global table quality | Data source quality - schema change | Data service | Tag Factory | Knowledge graph | Creation |
API | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported | Not supported | |
SAP Table | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
Salesforce | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
Lark Bitable data source | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | |
Neo4j | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported |
Vector database
Data source type | Offline integration | Real-time integration | Offline SQL development | Metadata collection | Real-time development | Global table quality | Table schema change | Data service | Label factory | Knowledge graph | Unstructured dataset | Guide |
Milvus | Supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Supported |