Data lineage for Flink deployments lets you trace the origin of your data, manage and optimize dataflows, quickly locate issues, and assess impact. This topic describes how to view data lineage from a deployment perspective and a metadata perspective.
Background
When the origin and change history of data are untraceable, it is difficult to ensure data quality and security, and data analysis and troubleshooting are inefficient. Data lineage addresses these challenges by describing the relationships formed as data is sourced, processed, transmitted, and consumed. These relationships include dataflows between metadata, dependencies, and the production and consumption relationships between metadata and stream or batch deployments. In Flink, you can view and analyze data lineage across multiple layers of deployments at both the table and column level. The following table describes the key benefits.
|
Benefit |
Description |
|
Improve data validation efficiency |
By tracing data lineage, you can understand every aspect of your data, including the products, databases, and tables involved in a deployment, the attributes and associations of table columns, and the data's origin, processing steps, transmission path, and final consumers. This knowledge helps you validate data credibility and accuracy, manage and analyze data more effectively, and work more efficiently. |
|
Improve troubleshooting efficiency |
When a data processing issue occurs, data lineage lets you trace the problem to its root cause for a quick resolution, preventing business losses and high labor costs. |
|
Improve data analysis efficiency |
When a data asset changes or contains errors, you can quickly identify affected online deployments and take corrective action to prevent incorrect decisions. |
|
Optimize data asset costs |
By analyzing data lineage, you can understand dataflow paths and dependencies. This allows you to optimize data processing pipelines, decommission long-idle services, improve data processing efficiency and quality, and reduce data costs. |
Data lineage model
The figure contains nodes (entities) and edges (relationships). Entities and relationships together form data lineage.
|
Element |
Description |
|
Node |
Each catalog, data table, and column is a data entity. In the lineage graph, an entity is represented as a node. Data lineage nodes include the following two types:
|
|
Edge |
The relationships between entities are defined by their upstream sources (producers) and downstream consumers. You can view the following lineage relationships:
|
Limitations
-
Data lineage is supported only within a single namespace. Cross-namespace data lineage is not supported.
-
To view data lineage from the metadata perspective, you must use a catalog. Viewing from the deployment perspective does not depend on a catalog.
-
Only SQL deployments support viewing and searching data lineage.
-
Data lineage is available only for SQL deployments that you have started at least once. When you stop a deployment, the system preserves its last known data lineage.
-
Only QueryOperation, SinkModifyOperation, and CreateTableAsTableOperation are supported. The system cannot track or display indirect lineage from sources like CDAS syntax, filter conditions, or join conditions.
View lineage from the deployment perspective
On the Deployments page, you can view specific information about deployment nodes and data nodes, as well as the dependencies between tables and the hierarchical relationships between tables and columns in the target deployment.
By default, the graph is centered on the deployment and displays three levels of lineage: the upstream table, the central deployment, and the downstream table. If you need to trace the lineage further upstream or downstream, you can click the plus sign to the left of the upstream table or to the right of the downstream table.
-
Log on to the Realtime Compute for Apache Flink console.
-
Click Console in the Actions column of the target workspace.
-
In the left-side navigation pane, choose , and then click the name of the target deployment.
-
On the Data Lineage tab, click Table Level or Column Level to view the corresponding data lineage.
-
Table Level
You can view information such as node type, connector, catalog name, database name, destination and source table names, as well as deployment ID, creation time, creator, last modification time, and last modifier.
In the Directed Acyclic Graph (DAG) of the deployment, data flows from the source node datagen through the processing node resource_setting_plan to the destination node blackhole. You can click a node to open a details panel and view the values of each column.
-
Column Level
You can view table columns, column types, the database name of the table, catalog name, and connector, as well as information about the deployment.
In the deployment's DAG, data flows from the datagen source node through the resource_setting_plan processing node to the blackhole sink node. Each node displays the connector type and column details, such as id and name, both of type VARCHAR(2147483647).
-
View lineage from the metadata perspective
If a table in a catalog is associated with many deployments, the lineage graph becomes cluttered. You can use features like zoom to navigate the graph. You can also turn on the Auto focus clicked node switch. When this feature is enabled, clicking a target deployment or table automatically centers the graph on that node.
-
Log on to the Realtime Compute for Apache Flink console.
-
Click Console in the Actions column of the target workspace.
-
In the left-side navigation pane, click Catalogs. Then, double-click the name of a table under a specific database of a catalog.
-
On the Data Lineage tab, click Table Level or Column Level to view the corresponding data lineage.
-
Table Level
You can see which deployments reference a specific table in the target catalog. When the table schema or data changes, you can quickly address the impact on referenced deployments by modifying their code or stopping them.
Double-click the target deployment node to view its ID and information about its creation and modification. To quickly navigate to the O&M page of the deployment, click the deployment name.
-
Column Level
You can click a target column and follow the light blue solid line to determine which deployment and table reference it, and which is the corresponding column. When the target column is deleted, or its name or attributes change, you can quickly find the referenced deployments, tables, and columns to handle the associated table columns accordingly.
The canvas displays associated intermediate query nodes (such as query-91) in the center. The properties panel on the right shows details for the selected column, such as node type, column type (for example, VARCHAR(2147483647)), and whether it can be null.
-
Search nodes or columns
When the lineage graph is complex, you can use a fuzzy search for a node or column name to quickly locate the target and view its information.
-
On the Data Lineage tab for the target deployment on the page, enter the target node name or column name (supported only at the column level) in the search box and press Enter.
The central node moves from the current deployment to the search target and is highlighted.
-
Double-click the target node or column name to view its lineage information.