View data profiling reports and records
After a data profiling task runs, you can view the data profiling reports for various field data types and the statistical distribution of the profiled fields. This topic describes how to view data profiling reports and records.
Prerequisites
The Data Quality module must be enabled to use the data profiling feature.
Permission description
Super administrators and operational administrators can view the profiling reports and records for all data tables. The owner of a data table can view only the profiling reports and records for the tables they own.
Project administrators can view the profiling reports and records for physical tables in the projects they manage.
Board administrators can view the profiling reports and records for logical tables in the boards they manage.
Regular members can view the profiling reports and records for which they have view permissions.
View data profiling results
On the Dataphin home page, choose Administration > Asset Checklist from the menu bar.
Click the Table tab to filter tables by type, such as physical tables, logical tables, physical views, logical views, and materialized views.
In the list of tables, click the target table name or the
icon in the Actions column to open the object details page.On the Object Details page, click the Data Profile tab to view the results of successful profiling tasks for which you have view permissions.
Profiling Record: Displays the profiling records of successfully run tasks that you have permission to view. This includes the profiling partition, profiling scope, number of profiled fields, and number of profiled records.
View Profiling Configuration: View the configuration of the profiling task.
View Log: View the operational log for the selected profiling record.
View data profiling reports
To ensure data security for sensitive data, original values are used for statistics if a data masking rule is configured for a field. However, the profiling report displays only the masked values.
After a profiling task runs successfully, you can view its report. The report shows the profiling results for each selected field across different profiling scenarios.
Field value distribution
This section provides statistics on the distribution of field values and generates a distribution chart. This chart helps you quickly understand the field value distribution and provides a reference for downstream data development and applications. This feature is supported for all data types, but the statistical metrics vary depending on the field data type.
Details about the field value histograms and bar charts are as follows:
If the field is a numeric type, a histogram that approximates the field's value distribution is displayed. The original record values are divided into 20 intervals. A line chart shows the number of records and the average value for each interval.
If the field is a text, date/time, or Boolean type, the Other Values category in the bar chart represents values outside the top 20 most frequent values, along with null value records.
If the field is a numeric type, the following statistics are calculated for the selected records: maximum (Max), minimum (Min), average value (Avg), number of null value records, number of unique value records, standard deviation, 25% quantile, median, and 75% quantile.
If the field is a text type, the following statistics are calculated for the selected records: maximum character length, minimum character length, average character length, number of null value records, and number of unique value records.
If the field is a date/time type, the following statistics are calculated for the selected records: maximum (Max), minimum (Min), number of null value records, and number of unique value records.
If the field is a Boolean type, the number of null value records is calculated for the selected records.
Null value statistics
This scenario helps you quickly identify if a field contains null values or other abnormal records. This can prevent related scheduling tasks from failing or affecting the accuracy of downstream data calculations. You can configure this profiling scenario for primary key fields or fields that cannot be null. This scenario is supported for all data types. Additionally, for numeric fields, the number of records with a value of 0 is also counted. For text fields, the number of empty string records is also counted.
Details about the null value statistics donut chart are as follows:
If the field is a numeric type, the chart shows statistics for the selected records, including the total number of profiled records, number of null value records, null value rate, number of records with a value of 0, rate of 0 values, and other values. The donut chart shows the distribution of these metrics.
If the field is a text type, the chart shows statistics for the selected records, including the total number of profiled records, number of null value records, null value rate, number of empty string records, empty string rate, and other values. The donut chart shows the distribution of these metrics.
If the field is a date/time or Boolean type, the chart shows statistics for the selected records, including the total number of profiled records, number of null value records, null value rate, and other values. The donut chart shows the distribution of these metrics.
If the field contains null values or empty strings, the following administration suggestions are provided:
If the field is a primary key (or a field that cannot be null) and is a numeric or text type, and the profiling result contains null value records, configure a field null value check monitoring rule for this field. This prevents downstream operations from being interrupted.
If the field is a primary key (or a field that cannot be null) and is a text type, and the profiling result contains null value records and empty string records, configure a field null value check or field empty string check monitoring rule for this field. This prevents downstream operations from being interrupted.
Unique value statistics
This section provides statistics on the number of unique value records for the field and the top 5 values with the highest number of duplicates. You can configure this profiling scenario for primary key fields that must have unique values or for fields with a high frequency of occurrence. Unique value statistics are not supported for Boolean fields.
If the field contains duplicate values, the following administration suggestions are provided:
If the field is a primary key and the profiling result contains duplicate values, configure a field value uniqueness monitoring rule for this field. This prevents downstream operations from being interrupted.
View data profiling records
Click the View Profiling Records button. The View Profiling Records panel opens.
The View Profiling Records panel shows each profiling record's name, type, status, and execution duration.
You can search for a target profiling record by its name or filter records by profiling status and profiling type.
In the Actions column of the target profiling record, you can perform the following operations.
Operation
Description
View Profiling Result
When the profiling task runs successfully, you can view the profiling report.
View Profiling Configuration
View the configuration information of the profiling task. If it is a manual profiling task, you can click the Profile Based on Current Configuration button at the bottom to quickly modify some information and start a new profiling task.
View Operational Log
View the operational log of the profiling task that corresponds to the selected profiling record.
Stop
For a manual or automatic profiling task that is running or waiting, you can stop the task.
Profile Based on Current Configuration
For a manual profiling task, you can use the configuration of this task to quickly pre-fill the settings and start a new profiling task. You cannot start a new task if the current one is in progress.