Use Ververica Flink in Dataphin to build real-time data pipelines and perform visual analytics.
Background information
A company uses MySQL as the storage database for its order system, with an oms_order table that stores all orders. For an upcoming marketing campaign, the company needs to track sales quantities by product type in real-time to quickly adjust the campaign strategy.
The following figure shows the data flow:
-
Ververica Flink tasks read data from the order system's MySQL database in real-time, aggregate sales quantities by product type, and write the results to another MySQL database for OLAP analysis.
-
BI tools, such as Quick BI, access the MySQL data for visual analytics.
Process guide
The following table describes the basic real-time R&D process in Dataphin:
|
Main Process |
Description |
Operation Guide |
|
Preparations |
Before you start, prepare the required cloud resources: set up an Alibaba Cloud account, activate and configure Dataphin, AccessKey, and Ververica Flink computing resources, optionally activate Quick BI, and prepare the data source. |
|
|
Project and Computing Source |
A real-time project is the basic unit for developing real-time tasks. You can associate it with Ververica Flink computing resources for real-time R&D. |
|
|
Real-time Meta Table |
Meta tables enable data management across different storage types. You can create and manage input, output, and dimension tables through meta tables. |
|
|
Real-time Task |
Create a Flink_SQL stream task to read from and write to the data sources mapped by meta tables. |
|
|
Task Operation and Maintenance |
After real-time R&D is complete, submit the task to the Operation Center to start or stop instances, view logs, monitor metrics, and configure alerts. |
|
|
Visual Analytics |
Optional. After the real-time instance starts, it continuously reads data from the source table, processes it with Ververica Flink, and writes results to the output table. You can use BI tools such as Quick BI to visualize the output data. |