Data Integration-Dataphin(Dataphin)-阿里云帮助中心

Data Integration is a simple and efficient data synchronization platform built on Dataphin. It provides powerful data preprocessing capabilities and high-speed, stable data synchronization across various heterogeneous data sources.

Get started in 5 minutes

Background

As big data adoption grows across industries, the demand for effective data integration has increased. Key requirements include efficiently configuring data synchronization tasks for a large number of tables, integrating with various heterogeneous data sources, performing light preprocessing on source data, and optimizing data synchronization tasks with features such as fault tolerance, rate limiting, and concurrency control.

Features

Note

If you purchased Dataphin after April 2020, the data synchronization capability has been upgraded to Data Integration.

Dataphin has upgraded its Data Integration capabilities, offering a simple, efficient, secure, and reliable data synchronization platform:

Improves the efficiency of data integration by supporting database migration (to quickly generate batch synchronization tasks) and one-click destination table creation (eliminating the need to manually create tables for data synchronized to MaxCompute). For more information, see Configure an Integration Task by Using Database Migration.
Supports process and transformation components to enable preprocessing of source data, such as cleansing, transformation, field masking, calculation, merging, distribution, and filtering. For more information, see Create an Integration Task by Using a Single Pipeline.
Supports both Dev-Prod and Basic development modes, so you can choose the mode that best fits your business scenario.
Supports quickly synchronizing a logical table created in Dataphin to a target database.
You can create custom components for data sources that are not natively supported to meet various business requirements for data synchronization. RDBMS database components connect through JDBC, while non-RDBMS database components require you to upload a JAR package.

Data Integration provides various components that you can drag and drop to create an offline single pipeline. It also supports the quick generation of batch synchronization tasks. For database migration, the supported sources are MySQL, SQL Server, and Oracle, and the supported destination is MaxCompute. Additionally, you can define custom components to meet your specific data synchronization needs.

Access Data Integration

Quick access (recommended)

On the Dataphin home page, click data import in the product pathway to go to the Data Integration page.

Standard access

On the Dataphin home page, choose Develop > Data Integration from the top menu bar to go to the Data Integration page.

Network connectivity

Data synchronization requires network connectivity between your data source and the Dataphin project workspace. For instructions, see Network Connectivity Solutions.

Use cases

Scenario	Description	Procedure
Build a synchronization task by using a pipeline script	Use an existing pipeline script to develop pipeline tasks for data synchronization.	Download the developed pipeline script. For more information, see Create an Integration Task by Using a Single Pipeline. Upload the pipeline script to create a pipeline. For more information, see Create an Integration Task by Using a Single Pipeline. Configure scheduling properties for the pipeline task. For more information, see Configure Offline Pipeline Task Properties. Submit or publish the pipeline task to the production environment. For more information, see Manage Publishing Tasks. Note If your development mode is Basic, you do not need to publish the pipeline task. Perform O&M and scheduling. For more information, see O&M center.
Build a synchronization task by using an offline single pipeline	An offline data pipeline transfers data between data sources by defining the source and destination. It provides a set of abstracted data input, data output, process, and transformation components, all built on a simplified intermediate data transfer format.	Configure a data source. For more information, see Supported Data Sources in Dataphin. Assemble and configure an offline single pipeline script. For more information, see Create an Integration Task by Using a Single Pipeline. To configure a batch synchronization task, see Configure an Integration Task by Using Database Migration. Submit or publish the pipeline task to the production environment. For more information, see Manage Publishing Tasks. Note If your development mode is Basic, you do not need to publish the pipeline task. Perform O&M and scheduling. For more information, see O&M center.
Build a synchronization task by using offline database migration	Database migration improves efficiency and reduces costs by letting you upload all tables from a MySQL, Oracle, or SQL Server database to MaxCompute in a single operation. This significantly reduces the configuration effort for your initial cloud migration.	Configure a data source. For more information, see Supported Data Sources in Dataphin. Assemble and configure an offline single pipeline script. For more information, see Create an Integration Task by Using a Single Pipeline. To configure a batch synchronization task, see Configure an Integration Task by Using Database Migration. Submit or publish the pipeline task to the production environment. For more information, see Manage Publishing Tasks. Note If your development mode is Basic, you do not need to publish the pipeline task. Perform O&M and scheduling. For more information, see O&M center.
Build a synchronization task by using a custom component	Data Integration lets you create a custom component for data sources that are not natively supported to meet various business requirements for data synchronization.	Create a custom component. For more information, see Create a Custom Offline Source Type. Create a data source based on the custom component. For more information, see Example of Developing a Custom Component. Create an offline single pipeline. For more information, see Create an Integration Task by Using a Single Pipeline. Submit or publish the pipeline task to the production environment. For more information, see Manage Publishing Tasks. Note If your development mode is Basic, you do not need to publish the pipeline task. Manage the pipeline task in the O&M center. For more information, see O&M center.