Data quality monitoring node

Click Add Table. In the Add Table dialog box, search for and select the target table to monitor.

• For a non-partitioned table, the entire table is monitored by default. You can skip this step.

• For a partitioned table, select the partition that you want to monitor. You can use scheduling parameters. Click Preview to verify that the partition filter expression is calculated correctly.

You can create new rules or import existing ones. By default, configured rules are enabled.

• Create a new rule

  Click Create Rule to create a data quality monitoring rule from a template or by using custom SQL. The following sections describe these methods.

  Method 1: From a system template

  The platform provides various built-in monitoring rules. You can use these rule templates to quickly create a data quality monitoring rule. The following figure shows the procedure.

  Note

  You can also find the required rule template in the system template list on the left and click +Use to create a rule.

  

  System rule template parameters

  Parameter	Description
  Rule Name	A custom name for the rule.
  Template	Defines the type of check to perform on the table. DataWorks provides a wide range of built-in templates for both table-level and field-level monitoring. For more information, see View built-in rule templates. Note The templates for average, sum, minimum, and maximum values apply only to numeric fields.
  Rule Scope	The scope to which the rule applies. For a table-level rule, the scope defaults to the current table. For a field-level rule, select a specific field.
  Comparison Method	Defines how the rule checks whether the table data meets expectations. Manual Settings: Manually define how the data output is compared against the rule. The available comparison methods vary depending on the rule template. The actual options available are shown in the UI. For Numeric Type results, you typically compare the output to a fixed value (expected value). The comparison methods include Greater Than, Greater Than or Equal To, Equal To, Unequal To, Less Than, and Less Than or Equal To. You can define a range for normal data (normal threshold) and abnormal data (error threshold). For Fluctuation results, you typically perform a range comparison. The comparison methods include Absolute Value, Raise, and Drop. You can define a range for normal data (normal threshold) and also define thresholds for data with anomalies (warning threshold) and data that does not meet expectations (error threshold) based on the degree of deviation. Intelligent Dynamic Threshold: You do not need to manually configure fluctuation thresholds or expected values. The system uses an intelligent algorithm to determine a reasonable threshold. If a data anomaly is detected, an alert is triggered or the task is blocked. Dynamic thresholds also support strong and weak rules. Note Only Custom SQL, Custom Range, and Dynamic Threshold data quality rules support the intelligent dynamic threshold comparison method.
  Monitoring Threshold	If you set Comparison Method to Manual Settings, you can specify the Normal threshold and Error Threshold. Normal threshold: If the check result meets this value, the data passes validation. Error Threshold: If the check result meets this value, the data fails to meet expectations. If the rule is a Fluctuation check, you must specify a Warning Threshold. Warning Threshold: When the data quality check result meets the value set here, it indicates that an anomaly exists in the data but it does not affect business operations.
  Retain problem data	If an enabled rule fails its check, the system automatically creates a table to store the identified problematic data. For more information, see Manage problematic data. Important This feature is currently supported for MaxCompute and Hologres tables. Only some data quality monitoring rules support this feature. For a list of rules that support retaining problematic data, see Appendix: List of rules that support retaining problematic data and their specifications. If the rule is Deactivate, problematic data will not be retained.
  Status	The Enable and Deactivate statuses control whether the rule runs in the production environment. Important When the status is set to Deactivate, the rule cannot trigger a test run and will not be triggered by associated scheduling tasks.
  Degree of importance	The importance of the rule in your business workflow. Strong rule: A critical rule. If an error alert occurs, the system blocks the associated scheduled task by default. Weak rule: A standard rule. If an error alert occurs, the system does not block the associated scheduled task by default.
  Configuration Source	The source of the rule configuration, which defaults to Data Quality.
  Description	You can add a description for the rule.

  Method 2: From a custom template

  Before you use this method, go to Data Quality > Quality Assets > Rule Template Library to create a custom rule template. You can then create a data quality monitoring rule based on that template. For more information, see Create and manage custom rule templates.

  The following figure shows how to create a data quality rule from a custom template.

  Note

  You can also find the required rule template in the custom template list on the left and click +Use to create a rule.

  

  Custom rule template parameters

  This section describes only parameters that are unique to custom rule templates. For information about other parameters, see the system rule template parameter descriptions.

  Parameter	Description
  Flag parameters	Defines the SET commands that need to be executed before the data quality check SQL is run.
  SQL	Defines the complete SQL check logic. The query must return a single numeric value. In the custom SQL, use square brackets to match the partition filter expression of the table. Example: SELECT count(*) FROM ${tableName} WHERE ds=$[yyyymmdd]; Note The system dynamically replaces the ${tableName} variable with the name of the monitored table. For more information about how to configure a partition filter expression, see . If you have created a quality monitoring rule for the table, when you configure a rule using this method, the Data Scope set in the quality monitoring settings will not take effect. The rule will use the WHERE clause in this SQL statement to determine which table partition to check.

  Method 3: From custom SQL

  This method lets you define custom data quality check logic for a table.

  

  Custom SQL parameters

  This section describes only parameters that are unique to custom SQL. For information about other parameters, see the system rule template parameter descriptions.

  Parameter	Description
  Flag parameters	Defines the SET commands that need to be executed before the data quality check SQL is run.
  SQL	Defines the complete SQL check logic. The query must return a single row and a single column, and the result must be a numeric value. In the custom SQL, use square brackets to match the partition filter expression of the table. Example: SELECT count(*) FROM <table_name> WHERE ds=$[yyyymmdd]; Note You must replace <table_name> with the name of the target table. This SQL statement determines which table is monitored. For more information about how to configure a partition filter expression, see . If you have created a quality monitoring rule for the table, when you configure a rule using this method, the Data Scope set in the quality monitoring settings will not take effect. The rule will use the WHERE clause in this SQL statement to determine which table partition to check.

• Import existing rules

  If monitoring rules for the target table already exist in the Data Quality module, you can import them to quickly clone the rules. If no rules exist, you must first create them in the Data Quality module. For more information, see Configure rules for a single table.

  Note

  This method supports importing multiple rules in a batch and allows you to configure monitoring rules at the field level.

  Click Import rules. You can search for and select the rules to import by rule ID or name, rule template, or associated scope (the entire table or specific fields).

  

Select the compute resources required to run the quality rule check. This specifies the data source in which the data quality monitoring task runs. By default, the data source of the monitored table is used.

The following table describes the categories of check exceptions.

Configure a policy to handle exceptions found by the rule check as needed:

• Do not ignore: Configure the system to stop the current node and set its status to failed when a specific exception category (for example, an error alert for a strong rule) is detected.

  Note

  • If the current node fails, its downstream nodes will not run. This blocks the production pipeline and prevents problematic data from spreading.

  • You can add multiple exception categories for detection.

  • This policy is typically used when an exception has a major impact and needs to block the execution of downstream tasks.

• Ignore: Ignore the exception and continue to run downstream nodes.

You can configure how to receive exception notifications, such as by email. When an exception occurs, the platform sends a notification using the specified method so you can promptly find and handle the exception.