Data source table

Section

Description

① Basic Information

This section displays basic information about the metadata, such as its name, environment, tags, and description. You can also perform the following actions:

For a data source table from Elasticsearch, you can search metadata, view asset details, view the production or development object, modify tags, add the asset to favorites, report a quality issue, apply for data source permissions, and export fields (table objects only).

• If the metadata has been deleted from the source system, its name is grayed out. You can click the icon next to the name to open the Metadata List page and delete the metadata asset.

• Search Metadata: In the search box in the upper-right corner, enter keywords such as a table name to find a data source table. You can also switch to view the details of other assets.

• View Asset Details: If the current object has been published to the Asset Catalog, you can go to the catalog details page to view the published information.

• View Production/Development Object: In the upper-right corner, click View Production/Development Object to switch to the object details in the corresponding production or development environment.

• Tags: This section displays the tags configured for the current asset. To modify tags, click Edit.

  • Each tag can be up to 128 characters in length.

  • You can configure a maximum of 20 tags for each asset object.

• Favorite: Click this button to add the asset to or remove it from your favorites. After you add an asset to your favorites, you can view the 30 most recently favorited assets on the My Footprints tab of the Asset List page for quick access. You can also view all your favorited assets in the Personal Data Center. For more information, see View and manage my favorite assets.

• Apply for Permissions: Click this button to go to the permission application page for the current data table. For more information, see Apply for, renew, and revoke table permissions.

• Report Quality Issue: This operation is not supported if the metadata has been deleted. You can use this feature to report data quality issues such as unstable data output or inaccurate data content to the responsible personnel. This helps ensure that issues are addressed promptly, improving asset usability and health. For more information about the configuration, see Add and manage issue lists.

  You must enable the data quality module to use this feature. You can report quality issues only for data source tables in the production environment. You can track the progress and resolution of reported issues in the issue list of the data quality module.

• Generate SELECT statement: Click this button to generate a query statement for the current table. You can choose whether to add escape characters. You can copy the query statement and run it on the ad hoc query or analysis page.

• View DDL statements: In the upper-right corner, click More and select View DDL statements to view the DDL statements for the current metadata asset in its data source. You can also select a Data Source Type and click Generate DDL statements. The system generates DDL statements for creating a table with the same structure as the current metadata for the specified data source type. If you select Automatically Add Escape Characters, the generated DDL statement automatically adds the appropriate escape character for the selected source type. This helps reduce issues such as misinterpretation of reserved keywords.

• Apply for Data Source Permissions: In the upper-right corner, click More and select Apply for Data Source Permissions to open the permission application page for the current data source. For more information, see Apply for, renew, and revoke data source permissions.

• Export Fields: In the upper-right corner, click More and select Export Fields to export the field information for the asset in CSV format. This allows developers or business personnel to quickly analyze and use the data.

② Details

This section displays the properties and field information for tables and views.

• Table Details: This section displays the business and technical properties of the metadata.

  • Business Properties: Includes information about the source system.

  • Technical Properties: Includes the source data source, table row count, highest security level (requires the purchase of the data security feature), whether the table is a lake table, lake table format, table storage mode, storage type (internal table or external table), storage size, and location information.

    • Table storage mode: If the data table is a lake table in Hudi format, this information can be collected from the table's source (data source). This information cannot be collected from StarRocks, SelectDB, and Doris data sources.

    • Location: The storage address of the current table. Example: hdfs://node01.exp5:8020/user/hive/warehouse.

      Note

      • For a view, only the source data source information is available in the technical properties.

      • Storage type information is not available for MySQL and Microsoft SQL Server data sources.

      • For an Oracle data table, the storage size is estimated based on the table row count and the average space used per row in bytes. This calculation does not include statistics for indexes, unused space, or fragmentation. Therefore, this estimate is lower than the actual space occupied.

      • If the data source table is a partitioned table, the row count information is for reference only. The system does not support collecting row count information for the main table of a partitioned table.

      • You can view the location information if the data source is Greenplum and the storage type is external table.

      • For an Amazon Redshift data source, you can view only the source data source and storage type information.

      • If the data source is Elasticsearch and the object is a Table (index), you can view only the document count and storage size in the technical properties.

• Field Information: This section displays the field information for the table or view, including the field name, data type, description, if the field is nullable, default value, and data classification/data security level. The data classification/data security level requires you to enable the data security module. If the data source is Elasticsearch and the object is a Table (index), you can view the field name and data type information.

  Search and Filter: You can search for fields by name or description, or filter fields by data classification or data security level. The filter feature requires you to enable the data security module.

• Partition Information: You can view the partition information of the data table. Multi-level partitions are displayed as a path, with levels separated by a forward slash (/). For example: ds=20221001/pt1=a/pt2=b.

  Note

  • The partition record count and storage size are for reference only.

  • You cannot view partition information if the data source is Elasticsearch.

③ Lineage & Impact

• The Lineage tab shows the lineage relationships between tables and fields.

  • Sources for table-level lineage include data synchronization tasks, computing tasks with custom lineage, and SQL-based computing and logical table tasks whose lineage can be automatically parsed by the system. For more information, see Table-level lineage.

  • Sources for field-level lineage include computing tasks with custom lineage, and SQL-based computing and logical table tasks whose lineage can be automatically parsed by the system. For more information, see Field-level lineage.

    Note

    The system parses table- and field-level lineage relationships for the development environment when a task is submitted, and for the production environment when the task is published. A single task submission or publication can parse up to 100,000 lineage relationships. If this limit is exceeded, the relationships are not recorded and are not displayed in the Asset Catalog.

• The Impact tab is divided into data table impact and integration and synchronization impact.

  • Data Table Impact: This section displays the direct downstream tables that reference the current table and the indirect tables that reference those direct tables. You can export data and show only direct downstream tables.

    • Export Data: You can export the data as an Excel file for business personnel to browse.

    • Show Only Direct Downstream Tables: If you select this checkbox, only the downstream tables directly affected by the current table are displayed. If you clear this checkbox, all downstream impact tables up to 15 levels deep are displayed. The direct downstream tables are considered the first level.

  • Integration and Synchronization Impact: This section displays the integration and synchronization tasks that use the current table and its downstream tables as a source. You can export data and show only the impact of the current table.

    • Export Data: You can export the data as an Excel file for business personnel to browse.

    • Show Only the Impact of the Current Table: If you select this checkbox, only the integration and synchronization tasks where the current table is an input are displayed. If you clear this checkbox, the integration and synchronization tasks that use the current table and all its downstream impact tables as inputs are displayed, up to 15 levels deep. The direct downstream tables are considered the first level.

④ Quality Overview

You must enable the data quality feature to view the rule check overview and the quality monitoring rule list for the current data table. You can click View Report Details or View Rule Details to open the corresponding page in the data quality module for more details. The quality overview is available only for data tables that support data quality monitoring. For a list of supported data sources, see Data sources supported by Dataphin.

You can create quality rules and view the quality overview only for data source tables in the production environment.

⑤ Data profiling

If you have enabled the data quality feature, you can initiate and view data profiling reports for supported data source tables. This allows you to quickly understand the data's general characteristics and assess its usability and potential risks. To enable automatic data profiling, go to Govern > Metadata Center > Profiling and Analysis and configure the settings. For more information about how to configure data profiling tasks, see Create a data profiling task. For a list of data sources that support data profiling, see Data sources and compute engines supported for data profiling.

⑥ Data preview

If you have permission to query the table, you can use the data preview feature. This preview shows the first 50 rows of data and includes only the fields for which you have SELECT permissions. For information about how to apply for query permissions, see Apply for, renew, and revoke table permissions.

You can search or filter the data by field, view single-row details, auto-fit column widths, expand or collapse field display names, and switch between row and column views. You can also click the sort icon next to a field to sort the data in Ascending or Descending order, or select No Sorting. Double-click a field value to copy it.

⑦ Asset information

This section displays basic and usage information for the data source table.

• Basic Information: Includes the environment, table type, last collection time, and collection task of the data source table.

  • Last Collection Time: The completion time of the last successful collection task for the table. The format is YYYY-MM-DD hh:mm:ss.

  • Collection Task: Information about the collection task that corresponds to the data source table.

• Usage Information: Includes the number of favorites and page views for the data source table.

  • Favorites: Displays the number of users who have favorited the data source table. This value is updated in real time.

  • Page views (PV): The total number of views for the data source table's details page. The count increases by one each time the page is refreshed and is updated in real time.

Number

Description

① Quick actions

• Search: You can search for expanded nodes on the current page by table name. Click the left or right arrows next to the search box to navigate between matched nodes. The system focuses on the first matched node by default.

• Show Only Direct Lineage: Click Filter and select Show Only Direct Lineage to display the upstream and downstream tables that have a direct lineage relationship with the current table.

  Note

  • Direct lineage refers to upstream or downstream assets that are read from or written to by using statements such as SELECT and INSERT.

  • Associated lineage refers to related assets that operations such as JOIN, GROUP BY, or WHERE conditions reference.

  • A table-level lineage graph displays the direct lineage identifier if any of its fields have direct lineage.

• Hide Data Source Tables: Click Filter and select Hide Data Source Tables to hide table objects from data sources and their downstream dependencies.

• Hide Uncollected Nodes: Lineage is parsed when tasks are submitted and published. If a data table is deleted outside of a Dataphin scheduled task, for example, through an ad hoc query or directly in the engine, Dataphin does not update the lineage. The table still appears in the lineage graph as an uncollected node. Select this option to hide tables that have been deleted or have not yet been collected.

② Legend

Table-level lineage supports various data tables, including physical tables, dimension logical tables, fact logical tables, aggregate logical tables, tag logical tables, physical views, materialized views, logical views, meta tables, mirrored tables, and data source tables.

③ Lineage graph

This area displays the complete lineage path. You can manually expand multiple levels of upstream or downstream dependencies and perform a fuzzy search by data table name. If a circular dependency exists, you cannot expand the node further and must trace the downstream path from the starting node.

• Central node: Displays the name of the current asset, which is marked with a Current Table label in the upper-left corner.

• Fields in the table: Hover over a node and click the drop-down arrow on the right to perform a fuzzy search for fields by name. Click a field name to go to the asset details page and view its lineage.

④ Object details

Hover over a table to view its details.

For a data source table, the details include its Name, Object Type, Data Source Type, Data Source, and Lineage Source. You can also perform the following actions: View Lineage, View DDL statements, and Apply for Permissions.

• If the metadata has been deleted from the source system, you can click the icon next to the name to open the Metadata List page and delete the metadata asset.

• View Lineage: Click this button to view the table-level lineage of the table.

• View DDL statements: If the data table is a physical table, physical view, or mirrored table, you can click this button to view the table's DDL statements.

• Apply for Permissions: This action is not supported if the metadata has been deleted. For an active metadata asset, you can click this button to open the table's permission application page.

Number

Description

① Quick actions

• Search: You can search for expanded nodes on the current page by field name. Click the left or right arrows next to the search box to navigate between matched nodes. The system focuses on the first matched node by default.

• Show Only Direct Lineage: Select this option to display only the upstream and downstream fields that have a direct lineage relationship with the current field.

  Note

  • Direct lineage refers to upstream or downstream assets that are read from or written to by using statements such as SELECT and INSERT. Calculation logic processed in a filter component with a WHERE condition or in a field calculation component with a built-in function also appears as direct lineage in the field-level lineage graph.

  • Associated lineage refers to related assets that operations such as JOIN, GROUP BY, or WHERE conditions reference.

• Hide Uncollected Nodes: Lineage is parsed when tasks are submitted and published. If a data table is deleted outside of a Dataphin scheduled task, for example, in an ad hoc query or directly in the engine, Dataphin does not update the lineage. The table still appears in the lineage graph as an uncollected node. Select this option to hide tables that have been deleted or have not yet been collected.

② Legend

Field-level lineage supports fields from compute source tables and data source tables.

③ Lineage graph

This area displays the complete lineage path. You can manually expand multiple levels of upstream or downstream dependencies and perform a fuzzy search by field name. If a circular dependency exists, you cannot expand the node further and must trace the downstream path from the starting node.

Central node: Displays the name of the current field and its table, which is marked with a Current Field label in the upper-left corner. You can perform a fuzzy search by field keyword to switch to the lineage graph of a different field.

④ Object details

Hover over a field to view its Name, Object Type, Data Source Type, Table, Table Owner, Data Type, Description, and Lineage Source. You can also perform the View Lineage action.

• Table: If a collection task is configured for the data source table, click the table name to open the details page for the table's data source.

• Apply for Permissions: For an active metadata asset, you can click this button to go to the table's permission application page.

• View Lineage: Click this button to view the lineage graph with this field as the central node.