Bulk Create Quality Rules

Prerequisites

You have published the data tables or metrics to the production environment. For more information, see Manage release tasks.

Permissions

• Super administrators and quality administrators can bulk configure quality rules, create and delete anomaly archive tables, and configure quality score weights.

• Quality owners can configure quality rules, create and delete anomaly archive tables, and configure quality score weights for the monitoring objects they own.

• Quality owners and regular users must have read permissions on data tables and data sources. To apply for permissions, see Apply for, renew, and revoke table permissions and Apply for data source permissions.

• The supported operations and permissions vary by object. For more information, see Quality rule operation permissions.

Validation Rules

When a data table is checked against quality rules, if it triggers a weak rule, the system sends an alert to help you promptly identify and address the issue. If it triggers a strong rule, the system blocks the associated task to prevent dirty data from flowing to downstream tasks. The system also sends an alert.

Bulk Add Quality Rules

Adding quality rules in bulk is an efficient way to apply the same rule to multiple objects at the table or field level. The configuration process is similar for different monitoring objects, with the main difference being how you select the objects. This guide uses a Dataphin table as an example.

1. On the Dataphin homepage, click Governance > Data Quality in the top navigation bar.

2. In the left-side navigation pane, click Quality rules. On the Quality rules page, click Add quality rule in the upper-right corner or click the icon and select Add by monitored object.

3. On the Add quality rule page, configure the parameters.

   1. Configure basic information

      Parameter	Description
      Rule name	A custom name for the quality rule, up to 256 characters. You can adjust the name for each object after you select the monitoring objects.
      Rule strength	You can select weak rule or strong rule. Weak rule: If you select Weak rule, an alarm is triggered but downstream task nodes are not blocked if the quality rule check fails. Strong rule: If you select Strong rule, an alarm is triggered when a quality rule check fails. If downstream tasks exist (such as code check scheduling and task trigger scheduling), these tasks are also blocked to prevent the spread of polluted data. If no downstream tasks exist (such as for periodic quality scheduling), only an alarm is triggered.
      Description	A custom description for the quality rule, up to 128 characters.
      Configuration method	Supports template-based creation and custom SQL. Create from template: Quickly create a quality rule by using a general-purpose system template or a custom business template. System template: The template has built-in parameters that you can configure. It is suitable for creating general-purpose rules. Custom template: The template has preset parameters that do not require configuration. It is typically used for creating rules that involve business logic. custom SQL: Define quality rules by using SQL. This method is suitable for complex scenarios. Only custom SQL templates support bulk configuration of quality rules. Note Data sources and real-time metatables do not support configuration.
      Rule template	The supported rule templates vary by monitoring object. For more information about templates, see Template type description. Dataphin tables and global data tables support templates for Integrity, Uniqueness, Timeliness, Validity, Consistency, Stability, and custom SQL. Integrity: Includes Field Null Value Check and Field Empty String Check. Uniqueness: Includes Field Uniqueness Check, Field Group Count Check, and Field Duplicate Count Check. Timeliness: Includes Time Function Comparison, Single-Table Time Field Comparison, and Two-Table Time Field Comparison. Validity: Includes Field Format Check, Field Length Check, Field Value Range Check, Code Table Reference Comparison, and Data Standard Code Table Reference Comparison (requires the data standards module). Consistency: Includes Single-Table Field Value Consistency Comparison, Single-Table Field Statistical Value Consistency Comparison, Single-Field Business Logic Consistency Comparison, Two-Table Field Value Consistency Comparison, Two-Table Field Statistical Value Consistency Comparison, Two-Table Business Logic Consistency Comparison, and Cross-Source Two-Table Field Statistical Value Consistency Comparison. Stability: Includes Table Stability Check, Table Fluctuation Check, Field Stability Check, and Field Fluctuation Check. custom SQL: Defines a rule by using a custom SQL script. Metrics support templates for Uniqueness and Stability. Uniqueness: Includes Field Group Count Check and Field Duplicate Count Check. Stability: Includes Field Stability Check and Field Fluctuation Check. Data sources support templates for Stability. Connectivity Monitoring: If a data source configured in Dataphin becomes unreachable due to network changes, incorrect credentials, or other reasons, tasks can fail. This rule monitors and sends alerts for connectivity changes. Table Structure Change: Changes to an upstream table's structure, such as renaming, deleting, or adding fields, can cause downstream tasks to fail. This rule monitors and sends alerts for table structure changes. This rule is supported only by specific data sources. For more information, see Data sources supported by Dataphin. Real-time metatables support templates for Consistency and Stability. Consistency: Includes Real-time and Offline Comparison and Real-time Cross-Link Comparison. Stability: Includes Real-time Statistical Value Detection.
      Rule type	The rule type depends on the template and is a fundamental attribute used for description and filtering.
      Object filtering	You can filter monitoring objects based on various criteria. Dataphin table: Filter data tables by project (for physical tables) or business segment (for logical tables), resource owner, and table type. You can select up to 100 projects or business segments. Global data table: Filter data tables by data source type, data source, and DB/Schema. For supported data sources, see Data sources supported by Dataphin. If a data source cannot connect to the Dataphin cluster, you must first perform metadata collection before configuring quality rules. For more information, see Create and manage metadata collection tasks. Metric: Filter metrics by data segment and summary logical table. Data source: Filter data sources by type. If you select the Connectivity Monitoring rule template, you can create quality monitoring rules for all data sources in Dataphin. All supported data sources can be tested for connectivity. If you select the Table Structure Change Monitoring rule template, select the data tables to check by DB/Schema. For information about data sources that support table structure change monitoring, see Data sources supported by Dataphin. Real-time metatable: Filter real-time metatables by project.
      Object selection	Field object: If you need to configure field-level monitoring rules, you can first select the data tables to monitor by table name, table owner, or quality owner, and then select the specific fields to monitor. Table object: You can configure table-level monitoring rules. This is available when you select the Stability-Table Stability Check or Stability-Table Fluctuation Check rule template for a data table, or the Stability-Table Structure Change Monitoring rule template for a data source. You can select data tables by table name, table owner, or quality owner.

   2. Click Next.

      If you click Cancel, none of the quality rules you configured will be added.

   3. Configure the rule (Skip this step for data sources.)

      Parameter	Description
      Benchmark table	The data table you selected in the Object selection step. Rule details are configured based on the fields of this table. For example, table A has id and name fields, table B has id and age fields, and table C has name and age fields. If you use table A as the benchmark table and select the id field for validation, the check passes for table B but fails for table C. Note You must configure a benchmark table when the monitoring object is a data table or a real-time metatable and the rule template has a complex configuration (meaning it requires other fields in addition to the check field). A benchmark table provides a convenient way to select fields when you need to bulk configure comparison fields for tables with different schemas. Use cases for benchmark tables: Bulk configuration is recommended for similar or identical requirements. If the requirements are completely different, using a benchmark table will cause an error during validation in the third step.
      Template configuration	When you select a quality rule template, its configuration information is displayed. To modify the configuration, see Quality rule templates.
      Rule configuration	The rule configuration varies depending on the selected rule template. For details about configuring rules for data tables, see Data table parameter configuration. For details about configuring rules for metrics, see Metric rule configuration. For details about configuring rules for real-time metatables, see Offline link comparison parameter configuration and Multi-link comparison parameter configuration. Special configurations are as follows: Filter verified table data: This is disabled by default. If you enable it, you can configure filter conditions for the verified table, such as partition filtering or regular data filtering. The filter conditions are appended to the validation SQL. If the verified table requires partition filtering, we recommend that you configure a partition expression in the scheduling configuration. After configuration, you can view the quality report for each checked partition. When you select the Consistency/Two-Table Field Statistical Value Consistency Comparison or Consistency/Cross-Source Two-Table Field Statistical Value Consistency Comparison rule template, you can enable Filter compared table data. If you enable this option, you can configure filter conditions for the compared table, such as partition filtering or regular data filtering. The filter conditions are appended to the validation SQL.
      Check configuration	After a quality rule runs, its result is compared with the anomaly check configuration. A check fails if the conditions are met, triggering processes such as alerts. The available metrics for anomaly checks are determined by the template and its configuration. Multiple AND/OR conditions are supported, but we recommend using fewer than three conditions. For more information, see Check configuration description.
      Archive configuration	This is Disabled by default. If you Enable it, you can archive anomalous data to a file or an archive table. After a quality check, you can download and analyze the archived data. For more information, see Anomaly archiving. Note Only Dataphin tables and global data tables support anomaly archiving.
      Business attribute configuration	The requirements for filling in business attributes depend on the configuration of the quality rule attributes. For example, if the field for the rule owner is set to custom input with a maximum length of 256 characters, you can enter up to 256 characters for this attribute when creating the quality rule. For more information, see Attribute information.
      Quality score configuration	A quality score evaluates check results to help you assess your data quality. For configuration details, see Quality score configuration. Note This configuration is supported only for Dataphin tables and global data tables.

   4. Click Next.

   5. Configure object details

      • Field-level detail configuration

        You can view the field check information for the selected data tables. You can also modify the rule name, quality owner, and quality score weight (only supported for Dataphin tables and global data tables). Additionally, you can edit the quality rule, delete the check object, and bulk manage quality owners and quality score weights.

        

        • Modify quality owner: Modify the quality owner for a selected monitoring object. You can also Bulk manage quality owners, which supports two modes: Add and Replace.

          Note

          If the object is already monitored, the quality owner specified here overwrites the existing one upon successful rule creation.

          • Add: Adds users or user groups to the current list of quality owners.

          • Replace: Overwrites the entire list of current quality owners with the specified users or user groups.

        • Modify quality score weight: Configure the quality score weight for the rule. This weight is used to calculate the monitoring object's quality score. You can set an integer from 1 to 10.

        • Edit: Modify the rule's Rule configuration, Check configuration, Business attribute configuration, and Quality score configuration.

        • Delete: Deletes the check object.

      • Table-level detail configuration

        You can view the check information for the selected data tables. You can also modify the rule name, quality owner, and quality score weight (only supported for Dataphin tables and global data tables). Additionally, you can edit the quality rule, delete the check object, and bulk manage quality owners and quality score weights. The operations for table-level configuration are the same as for field-level configuration. For more information, see Field-level detail configuration.

        

   6. Click Add rule to complete the configuration.

Bulk-Added Quality Rules List

After the quality rules are created in bulk, you can edit, run, schedule, and delete them from the rule configuration list.

Next Steps

After you configure scheduling in the rule list and click Done, you can view the new rules on the Dataphin table rules page. For more information, see Manage the monitoring object list.