Grouped metric validation rules
Grouped metric rules calculate an aggregate metric for each group and validate each result against a threshold. You can check fixed thresholds or percentage changes from historical samples for the same group to identify anomalies that a combined total may hide. If any group fails the check, the entire rule fails.
Configuration methods
For a system rule template, specify the grouping field in fields and the aggregate expression in templateParameters.metric. For an assertion, specify group_by inline in the metric expression in assertion. Use a template to configure the grouping field and metric separately. Use an assertion to write the complete metric expression directly.
| Rule | Template ID | Comparison | Rule result determination |
|---|---|---|---|
| Grouped metric with fixed thresholds | SYSTEM:field:group_by_metric_check:fixed | The current metric for each group against a fixed threshold | Check whether the metric value meets the fixed threshold specified by the comparison condition. For example, pass: when > 0 passes when the metric value is greater than 0; otherwise, the check fails. |
| Grouped metric historical volatility | SYSTEM:field:group_by_metric_check:flux | The current metric for each group against historical samples for the same group | Calculate the percentage change from the historical sample and evaluate it against the warn and fail thresholds. In this example, a change outside -5% to 5% satisfies the warning condition, and a change outside -10% to 10% satisfies the failure condition. |
The examples are rules fragments in a Spec configuration. For dataset configuration and other settings, see Data quality Spec configuration.
Grouped metric with fixed thresholds
Group records by category and check whether the sum of amount for each group is greater than 0. The following configurations express the same check: each result of sum(amount, group_by=category) must satisfy when > 0.
Use a system rule template
rules:
- templateId: SYSTEM:field:group_by_metric_check:fixed
name: Amount greater than zero for each category
fields:
- category
templateParameters:
metric: sum(amount)
pass:
- when > 0category in fields is the grouping field. sum(amount) in metric specifies the aggregate function and the field to aggregate. The corresponding metric expression is sum(amount, group_by=category).
Use an assertion
rules:
- assertion: "sum(amount, group_by=category)"
name: Amount greater than zero for each category
pass:
- when > 0The assertion specifies both the field to aggregate and the grouping field. You do not need to configure fields or templateParameters.metric. Apply the threshold to the sum for each group, rather than validating a combined total only once.
Grouped metric historical volatility
Group records by category and compare the current sum of amounts for each group with the sum for the same group seven days ago. The following configurations use the same historical offset and thresholds. A percentage change outside -5% to 5% satisfies the warn condition. A percentage change outside -10% to 10% satisfies the fail condition.
Use a system rule template
rules:
- templateId: SYSTEM:field:group_by_metric_check:flux
name: Compare amounts for each category with seven days ago
fields:
- category
templateParameters:
metric: sum(amount)
timeOffset: 7 days ago
warn:
- when not between -5% and 5%
fail:
- when not between -10% and 10%templateParameters.timeOffset specifies the historical comparison time. In this example, 7 days ago selects historical samples for the same group from seven days ago.
Use an assertion
rules:
- assertion: "change 7 days ago percent for sum(amount, group_by=category)"
name: Compare amounts for each category with seven days ago
warn:
- when not between -5% and 5%
fail:
- when not between -10% and 10%change 7 days ago percent for calculates the percentage change from historical samples from seven days ago. The assertion includes the historical offset, so you do not need to configure templateParameters.timeOffset.
Before you configure grouped metric historical volatility rules, prepare comparable historical samples for each group. Use the same aggregation method for the current and historical metrics.
Parameters
| Parameter | Applicable configuration | Description |
|---|---|---|
fields | System rule templates | The grouping field, such as category |
templateParameters.metric | System rule templates | The aggregate expression, such as sum(amount) |
templateParameters.timeOffset | Grouped metric historical volatility template | The historical offset, such as 7 days ago |
assertion | Assertions | Uses group_by=category in the metric expression to specify grouping. Percentage change expressions also include the historical offset. |
pass, warn, and fail | Templates and assertions | Use metric value thresholds for fixed-threshold rules and percentage thresholds for grouped metric historical volatility rules. |