Data Quality helps you immediately detect source data changes and dirty data from ETL (Extract, Transformation, and Load) processes. It automatically intercepts problematic tasks to stop dirty data from propagating downstream. This prevents tasks from producing unexpected data that can disrupt operations or affect business decisions. It also significantly reduces troubleshooting time and avoids the wasted resources and costs of re-running tasks.
Billing
The cost of running Data Quality rules consists of two parts:
-
DataWorks fees
You are charged on a pay-as-you-go basis for the number of Data Quality rule instances. For more information, see Billing for Data Quality instances.
-
Non-DataWorks fees
Data Quality rule checks generate verification SQL statements that are pushed to an engine for execution, which incurs engine fees. For billing details, refer to the documentation for each engine. For example, if you use MaxCompute on a pay-as-you-go basis, Data Quality rule checks generate MaxCompute engine fees. These fees are charged by MaxCompute and do not appear on your DataWorks bill.
Features
Data Quality supports quality checks for popular big data engines, including MaxCompute, E-MapReduce, Hologres, AnalyticDB for PostgreSQL, AnalyticDB for MySQL, and CDH. You can configure monitoring rules across multiple dimensions, such as completeness, accuracy, validity, consistency, uniqueness, and timeliness. By associating these rules with scheduling nodes, quality checks are automatically triggered after a task runs. This enables immediate detection of data issues. You can also set rule severity to control whether a task fails, which prevents the spread of dirty data and reduces the time and cost of data recovery.
The following section describes the features of each Data Quality module:
|
Parameter |
Description |
|
|
The Quality Dashboard provides a comprehensive overview of data quality in your workspace. It displays key metrics that require attention, trends and distribution of rule check statuses triggered by instance runs, lists of top tables with quality issues and their owners, and rule coverage. This helps quality managers quickly assess the overall data quality of the workspace and promptly address issues. |
||
|
Quality assets |
Displays a list of all configured quality rules. |
|
|
Data Quality lets you build your own rule template library to centrally manage common, custom monitoring rules. This improves rule configuration efficiency. |
||
|
Rule configuration |
Data Quality lets you configure quality monitoring rules by table or by template. |
|
|
Quality O&M |
The Quality Monitoring page displays all quality monitoring tasks created in the current workspace. |
|
|
Displays the rule check results of a quality monitoring task. After a task runs, you can view its details on this page. |
||
|
Quality analysis |
Data Quality lets you create report templates and add various metrics for rule configurations and runs. Based on the specified statistical period, delivery time, and subscription details, reports are automatically generated and sent. |
|
Limits
-
The supported regions vary by engine type:
Engine type
Supported regions
E-MapReduce
China (Hangzhou), China (Shanghai), China (Beijing), China (Zhangjiakou), China (Shenzhen), China (Hong Kong), Japan (Tokyo), Singapore, Malaysia (Kuala Lumpur), Indonesia (Jakarta), Germany (Frankfurt), and US (Silicon Valley).
Hologres
China (Hangzhou), China (Shanghai), China (Beijing), China (Zhangjiakou), China (Shenzhen), China (Hong Kong), China (Shanghai) Finance, Japan (Tokyo), Singapore, Malaysia (Kuala Lumpur), Indonesia (Jakarta), Germany (Frankfurt), US (Silicon Valley), and US (Virginia).
AnalyticDB for PostgreSQL
China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), and Japan (Tokyo).
AnalyticDB for MySQL
China (Shenzhen), Singapore, and US (Silicon Valley).
CDH
China (Shanghai), China (Beijing), China (Zhangjiakou), China (Hong Kong), and Germany (Frankfurt).
-
Before you can configure Data Quality rules for tables in E-MapReduce, Hologres, AnalyticDB for PostgreSQL, AnalyticDB for MySQL, or CDH, you must first collect metadata. For more information, see metadata collection.
-
After you configure Data Quality rules for tables in E-MapReduce, Hologres, AnalyticDB for PostgreSQL, AnalyticDB for MySQL, or CDH, the scheduling node that generates the table data must run on a resource group with network connectivity to properly trigger the Data Quality rule checks.
-
You can configure multiple Data Quality rules for a single table.
Scenarios
In offline data check scenarios, Data Quality uses the partition expression configured for a table to match the table partitions that a node generates each day. You associate the Data Quality rule with the scheduling node that produces the table data. When the task completes, the quality check is triggered. Dry runs do not trigger quality checks. You can set the rule severity to determine whether the node should fail and exit, which prevents the spread of dirty data. You can also configure alerts to receive immediate notifications and handle issues promptly.
Configure rules
-
Create rules: Data Quality allows you to create rules for individual tables. You can also use built-in rule templates to create rules for multiple tables in batch. For more information, see Configure monitoring rules by table and Configure rules: by template (batch).
-
Subscribe to rules: After creating a rule, you can subscribe to it to receive alert notifications for Data Quality rule checks. Supported notification methods include Email, Email and SMS, DingTalk Chatbot, DingTalk Chatbot @ALL, Feishu Group Robot, Enterprise WeChat Robot, and Custom Webhook.
NoteThe Custom Webhook method is available only in DataWorks Enterprise Edition.
Trigger rule checks
In Operation Center, after a scheduling node associated with a table finishes running its code logic, data quality monitoring is triggered. This process performs a rule check by executing a validation SQL query in the backend. The DataWorks platform then uses the strength of the data quality rule and the validation result to determine whether to terminate the task due to a rule failure. If the task is terminated, DataWorks blocks the execution of downstream nodes to prevent the spread of dirty data.
View check results
You can view Data Quality check results in the run log of a node in Operation Center or on the Data Quality run history page.
-
View the results in the run log of a node in Operation Center
-
Check the Instance Status. If the status is Quality monitoring check failed, this may indicate that the code ran successfully but the output data failed a strong rule check. This causes the task to fail and blocks downstream instances. On the Properties tab of the instance, you can see the Instance Status is set to Quality monitoring check failed.
-
On the task instance details page, click the Runtime Log tab and then the Data Quality Log button to view the data quality check details. For more information, see View periodic instances.
-
-
View the results on the Run History page.
On the run history page, you can search for the check details of a Data Quality monitoring task by table or node. For more information, see View monitoring tasks.