Enterprise-grade real-time data synchronization with Flink CDC
Enterprise-grade real-time data synchronization with Flink CDC
Traditional data integration pipelines often require separate systems for full and incremental synchronization. After the initial full synchronization, an additional merge process is required, complicating the architecture and increasing maintenance overhead. This solution uses Flink CDC to provide a unified, real-time integration pipeline for both full and incremental data.
Intended customers
Enterprises requiring data synchronization, data distribution, data warehouse updates, real-time analytics, and monitoring
Customers needing efficient real-time integration for large-scale datasets
Customers building a lakehouse or performing data ingestion
Benefits
Efficient and reliable data synchronization
Unlike traditional data integration pipelines, Flink CDC offers a unified solution for both full and incremental synchronization. A single Flink job can consistently synchronize the full dataset and incremental changes from the source to the downstream system. Additionally, Flink CDC uses an incremental snapshot algorithm that enables a seamless transition between full and incremental data reads without any extra configuration.
Simple and lightweight
Flink CDC uses log-based Change Data Capture (CDC) technology to unify reading full and incremental data. This reduces the number of components to maintain, simplifies the real-time pipeline, and lowers deployment costs.
Real-time and efficient
Flink CDC captures and synchronizes incremental changes from the source database in real time, ensuring data freshness and consistency. By using a stream processing model, Flink CDC efficiently handles large volumes of incremental data changes.
Flexible and scalable
Flink CDC easily scales to handle large-scale data processing scenarios, meeting growing data synchronization demands. Its simple API and connector design facilitate rapid development and integration, allowing flexible adaptation to various business requirements.
Flink CDC vs. traditional solutions
Traditional data synchronization solutions
Alibaba Cloud Flink CDC solution
Dual architecture for full and incremental data is difficult to maintain
Requires maintaining separate pipelines for full and incremental synchronization. Data is periodically merged to get the final result, but the merge job's frequency, limited by the scheduling system, reduces data timeliness. Furthermore, manually specifying the starting point for incremental synchronization makes it difficult to align with the completion point of the full synchronization, compromising data consistency.
Cost and maintenance
Unified full and incremental architecture simplifies maintenance
A unified framework for reading full and incremental data simplifies maintenance. Alibaba Cloud provides a rich set of automated O&M tools, allowing users to easily monitor system status without focusing on the underlying servers and infrastructure. The solution also supports horizontal scaling, which effectively reduces operational pressure.
Requires manual server resource management
When Debezium and Canal run as standalone instances without external orchestration, server resources must be managed manually. Their horizontal scaling capabilities depend on the deployment architecture and supporting components.
Stability and elasticity
Automatic scaling and high availability
The serverless-based service supports automatic scaling for jobs, dynamically adjusting resources based on real-time demand to handle different workloads.
Limited data source support and transformation capabilities
The ecosystem of traditional data integration tools offers limited support, requiring specific incremental synchronization services for different data sources. Data transformation capabilities are also limited and difficult to use.
Ecosystem and transformation
Rich ecosystem and powerful transformation capabilities
The solution supports a rich ecosystem of upstream and downstream systems, including Kafka, Apache Paimon, StarRocks, and Hologres, and also supports custom connectors. It also provides powerful data transformation capabilities, enabling various transformation operations during data synchronization through a CDC YAML job.
Architecture and deployment
Synchronizing RDS for MySQL to Apache Paimon
Flink CDC supports a rich ecosystem of upstream and downstream systems, such as Kafka, Apache Paimon, StarRocks, and Hologres. This solution uses ApsaraDB RDS for MySQL as the data source and Realtime Compute for Apache Flink to quickly build a real-time data lakehouse on Object Storage Service (OSS) using Apache Paimon as the storage format.
2860742
This solution configures a Flink CDC YAML job in Realtime Compute for Apache Flink to merge sharded database and table data from ApsaraDB RDS for MySQL into a single table in the data lakehouse, and to perform full database synchronization for other databases into the Apache Paimon data lake. The system automatically detects and synchronizes any modifications to the source table schema in real time to the downstream Apache Paimon data lake.
60 minutes
¥10.00(This is an estimated cost assuming you use the resource specifications from this topic's example for up to 60 minutes of runtime. Your actual costs may vary depending on factors like your traffic usage. The price shown in the console and on your final bill is final.)
Realtime Compute for Apache FlinkApsaraDB RDS for MySQLObject Storage Service
Use cases
Common use cases
Data distribution
For data distribution, Flink CDC efficiently and reliably transfers data from the source to downstream systems such as Apache Paimon, Kafka, and StarRocks.
Data ingestion into lakes and warehouses
Flink CDC integrates scattered and heterogeneous data sources into a data lakehouse in real time, eliminating data silos and enhancing data consistency and usability.
Stream integration
Flink CDC supports the YAML API for transformations such as selection, filtering, and custom functions. This facilitates stream-based data cleansing.
Free trial
Try Alibaba Cloud products for free
960470
951269
942060