Create and manage custom rule templates

Usage notes

Changes to a custom rule template affect only tasks that use the template after the modification. Existing tasks that use the original template are unaffected.

Create a custom rule template

1. Log on to the DataWorks console. In the target region, click Data Governance > Data Quality in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Quality.

2. In the left-side navigation pane, choose Quality Assets > Rule Template Library.

3. Click the icon and select Create Folder.

4. In the Create Folder dialog box, enter a Name, select a destination folder, and then click Confirm.

5. In the left-side directory, right-click the folder name and select Create Rule Template.

   You can also Rename or Delete the folder.

6. In the Create Rule Template panel, configure the parameters.

   The parameters to configure vary based on the Template Type. Click the corresponding tab to view the parameters. After a rule template is created, you cannot change its template type.

   Custom SQL

   Parameter	Description
   Template Name	Enter a name for the custom rule template.
   Template Type	Select Custom SQL. The check logic is defined by using an SQL statement.
   Set Flag	Enter any set statements to execute before the SQL query. Note Enter only the content that follows set. Separate multiple statements with commas (,). Do not add a semicolon (;) at the end of each statement.
   Check Type	The supported check types are Numeric Type and Fluctuation Rate Type.
   Verification Method	The available verification methods vary based on the selected check type: When the Numeric Type is Check Type, the only available verification method is Compare with a specified value. The SQL query must return a single value, such as a result from a COUNT or SUM operation, for comparison with the specified value. When the Fluctuation Rate Type is Check Type, the following verification methods are available: 7-day average fluctuation 30-day average fluctuation 1-day period comparison 7-day period comparison 30-day period comparison 7-day variance fluctuation 30-day variance fluctuation 1-, 7-, and 30-day fluctuation detection Previous-period comparison
   Custom SQL	Enter the SQL query. You can use the ${tableName} variable to represent the table name. Note The result of the Custom SQL query must be a single value (one row and one column) for comparison with a specified value.
   Quality Dimension	Select the quality dimension to which the rule template belongs, such as completeness, accuracy, or uniqueness. You can select only one quality dimension.
   Location	Select the folder in which you want to store the custom rule template.
   Only this workspace is visible	If you select this option, the rule template is visible only in the current workspace. You cannot change this setting after the rule template is created.

   Custom Script

   Parameter	Description
   Template Name	Enter a name for the custom rule template.
   Template Type	Select Custom Script. The check logic is defined by using a rule script in YAML format.
   Spec Content	Enter the rule definitions in YAML format. The syntax is the same as that of rules in a Data Quality Spec. Example: - assertion: row_count > 0. For more information about how to write rules, see Data Quality Spec configuration guide.
   Quality Dimension	Select the quality dimension to which the rule template belongs, such as completeness, accuracy, or uniqueness. You can select only one quality dimension.
   Location	Select the folder in which you want to store the custom rule template.
   Only this workspace is visible	If you select this option, the rule template is visible only in the current workspace. You cannot change this setting after the rule template is created.

7. Click Confirm.

Manage rule templates

Apply a custom rule template

When you add a monitoring rule, you can use a custom rule template.

1. Log on to the DataWorks console. In the target region, click Data Governance > Data Quality in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Quality.

2. Configure monitoring rules for a specific table.

   In the left-side navigation pane, choose Rule Setting > Configure by Table to go to the Configure by Table page.

   1. In the Connection list on the left, select the database where the target table resides.

   2. Filter the tables by database type, database, or table name. Click the name of the target table or click Actions in the Rule Management column to open the table quality details page.

      This page displays all configured quality monitoring schedules and rules for the current table. You can filter rules based on whether they are associated with a quality monitoring schedule and define how unassociated rules are run.

      The page has two tabs: Rule Management and Quality Monitoring. On the Rule Management tab, you can click New Monitoring Rule to create a rule. For existing rules, you can Modify or Delete them, and view the Operation Log.

3. On the Table Quality Details page, go to the Rule Management tab, select a quality monitoring schedule and then click Create Rule. This action opens the rule configuration page.

4. Create a Data Quality rule from a custom rule template.

   When you use a custom rule template whose template type is Custom SQL, its configurations, such as the FLAG parameter parameters and the SQL, are automatically populated. You can customize the Rule Name and configure monitoring thresholds based on the check type. For example, for a numeric rule, you need to define normal and critical thresholds. For a fluctuation rule, you also need to define a warning threshold.

   Click the + Custom Rule Template tab. In the Custom Rule Template list on the left, find the target template and click + Use next to it. This action automatically populates the form on the right with the template's configurations. Modify the parameters as needed.

   If the template type of the selected template is Custom Script, a script editor is displayed on the right, and a rule script that references the template is automatically generated. The script contains the template ID (templateId) and the rule name (name). You do not need to configure monitoring thresholds. You can modify the content, such as the rule name, as needed.

   For descriptions of the parameters, see Create a custom rule template.

5. Click Confirm.

Related documents

You can use a custom rule template when using the Configure by Table feature. For more information, see Configure monitoring rules for a single table.