View built-in rule templates for DataWorks Data Quality-DataWorks(DataWorks)-阿里云帮助中心

Data Quality in DataWorks offers a variety of built-in monitoring templates. This topic describes the verification logic of Data Quality and the available built-in monitoring rule templates.

Built-in rule templates

Use built-in rule templates to quickly set up monitoring rules for a single table or multiple tables in bulk. For details, see Create a monitoring rule for a single table and Create monitoring rules for multiple tables in bulk.

Template category	Template name	Description
Number of rows	Fixed value	Checks that the row count matches a fixed value.
	Table not empty	Checks that the table contains at least one row.
	1-day difference	Checks the daily change in row count. Note The baseline is the previous day's partition row count. The system compares the current day's collected row count against this baseline and calculates the difference.
	Change in table row count since the previous epoch	The baseline is the partition row count from the previous scheduling cycle. The system compares the current day's collected row count against this baseline and calculates the difference.
	1, 7, 30 days and 1st of month volatility	Compares the current row count with counts from 1 day ago, 7 days ago, 30 days ago, and the first day of the current month. Calculates volatility and compares it against thresholds. Triggers an alert if any threshold is breached.
	1, 7, 30 day volatility	Compares the current row count with counts from 1 day ago, 7 days ago, and 30 days ago to calculate volatility. Note Compares the current row count with the previous day's sample, the 7-day-ago sample, and the 30-day-ago sample. Calculates volatility and compares it against thresholds. Triggers an alert if any volatility exceeds the threshold.
	1-day volatility	Uses the previous day's row count as the baseline, compares it with the current day's collected row count, and calculates volatility. Triggers an alert if the threshold is breached.
	30-day volatility	Uses the row count from 30 days ago as the baseline, compares it with the current day's collected row count, and calculates volatility. Triggers an alert if the threshold is breached.
	7-day volatility	Uses the row count from 7 days ago as the baseline, compares it with the current day's collected row count, and calculates volatility. Triggers an alert if the threshold is breached.
	Table rows (intelligent dynamic threshold)	When Intelligent Dynamic Threshold is enabled, you do not need to manually configure volatility thresholds or expected values. The system automatically determines reasonable thresholds using intelligent algorithms and triggers alerts or blocks on anomalies.
	30-day average volatility	Monitors row count fluctuations. The baseline is the average row count over the past 30 days, calculated as (sum of daily row counts over 30 days) / 30.
	7-day average volatility	Monitors row count fluctuations. The baseline is the average row count over the past 7 days, calculated as (sum of daily row counts over 7 days) / 7.
	Upper cycle volatility	Uses the partition row count from the previous scheduling cycle as the baseline, compares it with the current day's collected row count, and calculates volatility.
	Custom range of table rows	Lets you define the comparison method and threshold range for table row count.
Condition match rate	Condition match rate, custom range	Lets you define the comparison method and threshold range for the match rate of filter conditions.
Table size	Table size, fixed value	The storage size of the table (in bytes).
	Table size, 1-day difference (bytes)	The difference in table storage size compared to 1 day ago (in bytes).
	Table size, upper cycle difference	The difference in table size compared to the previous scheduling cycle (in bytes).
	Table size, 1-day volatility	Monitors table size fluctuations. Compares the current table size with the previous day's sample, calculates volatility, and compares it against thresholds. Triggers an alert if any threshold is breached. For example, if you set the orange threshold to 5% and the red threshold to 10%, an orange alert is triggered when the volatility exceeds 5% but does not exceed 10%, and a red alert is triggered when the volatility exceeds 10%.
	Table size, 30-day volatility	Monitors table size fluctuations. Compares the current table size with the sample from 30 days ago, calculates volatility, and compares it against thresholds. Triggers an alert if any threshold is breached.
	Table size, 7-day volatility	Monitors table size fluctuations. Compares the current table size with the sample from 7 days ago, calculates volatility, and compares it against thresholds. Triggers an alert if any threshold is breached.
	Table size (intelligent dynamic threshold)	When Intelligent Dynamic Threshold is enabled, you do not need to manually configure volatility thresholds or expected values. The system automatically determines reasonable thresholds using intelligent algorithms and triggers alerts or blocks on anomalies.
Null value rows	Null count, fixed value	Compares the null count of the column against a fixed value. Note Null values are determined by converting to the SQL `IS NULL` expression.
Null value rows	Null count equals 0	Checks whether the null count is 0.
Null count / total rows	Null count / total rows, fixed value	Compares the ratio of null count to total row count against a fixed value. Note The fixed value is a decimal.
Format validation	Regex validation	Checks whether the column values match a regular expression.
	Date format validation	Checks whether the column values match a date format.
	Email format validation	Checks whether the column values match an email format.
	ID card format validation	Checks whether the column values match an ID card format.
	Phone number format validation	Checks whether the column values match a mobile phone number format.
	Currency format validation	Checks whether the column values match a currency format.
	Numeric format validation	Checks whether the column values match a numeric format.
	Telephone number format validation	Checks whether the column values match a telephone number format.
Duplicate value rows	Duplicate count, fixed value	The duplicate count is the total row count minus the distinct count. Compares the duplicate count against a fixed value.
Duplicate value rows	Column duplicate count equals 0	Checks whether the column duplicate count is 0.
Multi-column unique values	Multi-column duplicate count equals 0	Checks whether the multi-column duplicate count is 0.
Duplicate count / total rows	Duplicate count / total rows, fixed value	Compares the ratio of duplicate count to total row count against a fixed value.
Unique value count	Unique value count, fixed value	Compares the distinct count against an expected number (fixed value check).
	Unique value count, 1, 7, 30 day volatility	Compares the distinct count with samples from 1 day ago, 7 days ago, and 30 days ago to calculate volatility.
	Unique value count (intelligent dynamic threshold)	When Intelligent Dynamic Threshold is enabled, you do not need to manually configure volatility thresholds or expected values. The system automatically determines reasonable thresholds using intelligent algorithms and triggers alerts or blocks on anomalies.
Unique value count / total rows	Unique value count / total rows, fixed value	Compares the ratio of unique value count to total row count against a fixed value.
Minimum value	Minimum value, 1, 7, 30 day volatility	Retrieves the minimum value of the column and compares it with samples from 1 day ago, 7 days ago, and 30 days ago. Calculates volatility and triggers an alert if any threshold is breached.
	Minimum value (intelligent dynamic threshold)	When Intelligent Dynamic Threshold is enabled, you do not need to manually configure volatility thresholds or expected values. The system automatically determines reasonable thresholds using intelligent algorithms and triggers alerts or blocks on anomalies.
	Minimum value, 1-day volatility	Retrieves the minimum value of the column, compares it with the value from 1 day ago, and calculates volatility against the threshold.
	Minimum value, upper cycle volatility	Retrieves the minimum value of the column, compares it with the value from the previous scheduling cycle, and calculates volatility. Triggers an alert if the threshold is breached.
	Column minimum value, custom range	Lets you define the comparison method and threshold range for the column minimum value.
Maximum value	Maximum value, 1, 7, 30 day volatility	Retrieves the maximum value of the column and compares it with samples from 1 day ago, 7 days ago, and 30 days ago. Calculates volatility and triggers an alert if any threshold is breached.
	Maximum value (intelligent dynamic threshold)	When Intelligent Dynamic Threshold is enabled, you do not need to manually configure volatility thresholds or expected values. The system automatically determines reasonable thresholds using intelligent algorithms and triggers alerts or blocks on anomalies.
	Maximum value, 1-day volatility	Retrieves the maximum value of the column, compares it with the value from 1 day ago, and calculates volatility against the threshold.
	Maximum value, upper cycle volatility	Retrieves the maximum value of the column, compares it with the value from the previous scheduling cycle, and calculates volatility. Triggers an alert if the threshold is breached.
	Column maximum value, custom range	Lets you define the comparison method and threshold range for the column maximum value.
Average value	Average value, 1, 7, 30 day volatility	Retrieves the average value of the column and compares it with samples from 1 day ago, 7 days ago, and 30 days ago. Calculates volatility and triggers an alert if any threshold is breached. Note The average value of the column is compared with the average value from the previous day, 7 days ago, and 30 days ago, respectively.
	Average value (intelligent dynamic threshold)	When Intelligent Dynamic Threshold is enabled, you do not need to manually configure volatility thresholds or expected values. The system automatically determines reasonable thresholds using intelligent algorithms and triggers alerts or blocks on anomalies.
	Average value, 1-day volatility	Retrieves the average value of the column, compares it with the value from 1 day ago, and calculates volatility against the threshold.
	Column average value, custom range	Lets you define the comparison method and threshold range for the column average value.
Sum value	Sum value, 1, 7, 30 day volatility	Retrieves the sum of the column and compares it with samples from 1 day ago, 7 days ago, and 30 days ago. Calculates volatility and triggers an alert if any threshold is breached.
	Sum value (intelligent dynamic threshold)	When Intelligent Dynamic Threshold is enabled, you do not need to manually configure volatility thresholds or expected values. The system automatically determines reasonable thresholds using intelligent algorithms and triggers alerts or blocks on anomalies.
	Sum value, 1-day volatility	Retrieves the sum of the column, compares it with the value from 1 day ago, and calculates volatility against the threshold.
	Sum value, upper cycle volatility	Retrieves the sum of the column, compares it with the value from the previous scheduling cycle, and calculates volatility. Triggers an alert if the threshold is breached.
	Column sum value, custom range	Lets you define the comparison method and threshold range for the column sum value.
Enum values	Unmatched enum unique value count, fixed value	Compares the distinct count of unmatched enum values.
	Unmatched enum row count, fixed value	The row count of unmatched enum values in the column, compared against a specified fixed value.
	Unmatched enum row count equals 0	The expected normal state: the fixed value and comparison method are set to = 0. An alert is triggered whenever the unmatched enum row count is > 0.
	Enum values, custom, fixed value	Lets you customize enum-related statistics such as counts, row counts, comparison methods, and thresholds.
Discrete values	Discrete value (group count per value), fixed value	After a GROUP BY operation, the count of each group is compared against a fixed value.
	Discrete value (number of groups), fixed value	After a GROUP BY operation, the number of groups is compared against a fixed value.
	Discrete value (number of groups) (intelligent dynamic threshold)	When Intelligent Dynamic Threshold is enabled, you do not need to manually configure volatility thresholds or expected values. The system automatically determines reasonable thresholds using intelligent algorithms and triggers alerts or blocks on anomalies.
	Discrete value (group count per value) (intelligent dynamic threshold)	When Intelligent Dynamic Threshold is enabled, you do not need to manually configure volatility thresholds or expected values. The system automatically determines reasonable thresholds using intelligent algorithms and triggers alerts or blocks on anomalies.
	Discrete value (number of groups), 1-day volatility	After a GROUP BY operation, the number of groups is compared with the sample from 1 day ago to calculate volatility.
	Discrete value (number of groups and group count per value), 1, 7, 30 day volatility	After a GROUP BY operation, the number of groups and the count of each group are compared with samples from 1 day ago, 7 days ago, and 30 days ago to calculate volatility.

Note

Table size rule templates are not supported for EMR tables.

Calculation description

Volatility calculation: The volatility formula is (Current value - Baseline value) / Baseline value. Current value is the actual metric value of the current check, and Baseline value is dynamically determined by the template you select. For example:

For N-day volatility templates, the baseline is the metric value from N days ago for the same rule.
For N-day average volatility templates, the baseline is the arithmetic average of the metric values over the past N days.
For upper cycle volatility templates, the baseline is the metric value from the previous scheduling cycle.

Verification logic

Data Quality supports three verification methods: fixed value comparison, volatility comparison, and dynamic threshold.

Verification method	Verification logic
Fixed value comparison	Evaluates the verification expression and returns a Boolean value. The following comparison operators are supported: `>`, `<`, `>=`, `<=`, and `!=`. If the result is `true`, the check passes. Otherwise, a red alert is triggered.
Volatility comparison	This method monitors the magnitude of metric changes and supports three comparison types: Absolute value: Focuses on the magnitude of change regardless of direction. For example, an alert is triggered if the row count volatility exceeds 10%. Rise: Focuses only on abnormal increases. For example, when monitoring costs, you may only care about significant cost increases. Drop: Focuses only on abnormal decreases. For example, when monitoring order volume, you may primarily care about sudden drops. The system determines the alert level based on the orange threshold and red threshold you configure.
Dynamic threshold	You do not need to manually set thresholds. The system automatically detects metric correctness in real time using algorithmic models. An alert is triggered if the metric exceeds a reasonable fluctuation range.

Appendix 1: Previous cycle baseline description

The previous cycle task is determined by excluding checks with the same business date, sorting by business date in descending order, and if the business dates are the same, further sorting by run time in descending order. The first task in this sorted order is used as the baseline. See the following table for details:

Schedule type

Business date

Baseline comparison method

FAQ

Normal daily scheduling scenario

Historical business dates:

2024-06-01
2024-06-02
2024-06-03
2024-06-04
2024-06-05

When the scheduled task with business date 2024-06-06 starts verification, the check result from 2024-06-05 is used as the baseline for comparison.

Backfill data scenario:

Background:

Tasks from 2024-06-01 to 2024-06-05 were executed normally. After the scheduled instance on 2024-06-05 finished, a backfill was performed for the instance with business date 2024-07-01. When the scheduled task with business date 2024-06-06 starts verification, which date is used as the baseline for comparison?

Conclusion:

The scheduled task with business date 2024-06-06 uses the check result from business date 2024-07-01 as the baseline for comparison. Because the business date 2024-07-01 is later, all subsequent daily scheduled tasks will use 2024-07-01 as the baseline until the scheduled instance for 2024-07-02 finishes execution.

Normal hourly scheduling scenario

Historical business dates:

2024-06-01
2024-06-02
2024-06-03

Tasks run on an hourly schedule with 3 executions per day.

When the hourly scheduled task with business date 2024-06-04 starts verification, the last check result from business date 2024-06-03 is used as the baseline for comparison.

Hourly scheduling scenario:

Background:

From 2024-06-01 to 2024-06-03, three hourly scheduled instances ran each day, and the first scheduled instance on business date 2024-06-04 also ran normally. When the second scheduled instance on business date 2024-06-04 starts verification, which date is used as the baseline for comparison?

Conclusion:

When the second scheduled instance on business date 2024-06-04 starts verification, it excludes the first check from the same day (2024-06-04) and uses the last check result from business date 2024-06-03 as the baseline for comparison.

Appendix 2: N-day-ago sample value for hourly tasks

When an hourly task retrieves the N-day-ago sample, the system sorts the results from day N by run time (not scheduled time) in descending order and uses the first instance's result as the sample value for comparison with the current result. See the following table for details:

Schedule type

Business date

Volatility comparison method

FAQ

Normal hourly scheduling scenario

Historical business dates:

2024-06-01
2024-06-02
......
2024-06-08

Tasks run on an hourly schedule with 3 executions per day.

For a 7-day volatility comparison, when the hourly scheduled task with run date 2024-06-08 starts verification, the last sample value from run date 2024-06-01 is used.

Hourly scheduling scenario:

Background:

From 2024-06-01 to 2024-06-08, three hourly scheduled instances ran each day. When the second scheduled instance on run date 2024-06-08 starts verification, which day's and which scheduled instance's result is used as the sample value for the 7-day volatility comparison?

Conclusion:

When the second scheduled instance on run date 2024-06-08 starts verification, the last check result from run date 2024-06-01 is used as the sample value for the 7-day volatility comparison.