Create data source quality rules-Dataphin(Dataphin)-阿里云帮助中心

Prerequisites

You must add a monitoring object before you can configure a quality rule. For more information, see Add a monitoring object.

Limitations

You can create a quality rule for any data source in Dataphin. Connectivity tests are supported for all data sources, but table structure change monitoring is available only for specific data sources. For more information, see Supported data sources in Dataphin.

Permissions

Super administrators, quality administrators, data source owners, and users with a custom global role that includes the Quality Rules - Management permission can configure schedules and alerts for quality rules.
Quality owners and regular users need synchronous read permissions for the data source. To apply for data source permissions, see Apply for data source permissions.
The supported operations vary by object. For more information, see Quality rule operation permissions.

Trial run vs. run

A trial run simulates the execution of a quality rule to verify its correctness and performance. Trial run results do not appear in the quality report. A run executes the quality rule at the scheduled time, and its results appear in the quality report.

Configure a quality rule

In the Dataphin console, choose Governance > Data Quality from the top navigation bar.
In the left-side navigation pane, click Quality Rules. On the Data Source tab, click the name of the target object to go to the Quality Rule Details page.
On the Quality Rule Details page, click Create Quality Rule.

In the Create Quality Rule dialog box, configure the parameters.

Parameter		Description
Basic information	Rule name	Enter a custom name for the quality rule. The name can be up to 256 characters long.
	Rule strength	Valid values are weak rule and strong rule. Weak rule: If you select weak rule, an alert is triggered when the quality check fails, but downstream task nodes are not blocked. Strong rule: If you select strong rule, an alert is triggered when the quality check fails. If there are downstream tasks (such as in a code-check or task-triggered schedule), they are blocked to prevent bad data from spreading. If there are no downstream tasks (such as in a periodic quality schedule), only an alert is triggered.
	Description	Enter a custom description for the quality rule. The description can be up to 128 characters long.
	Rule template	Only Stability is supported. This template includes Data source connectivity monitoring and Table structure change monitoring. Connectivity monitoring: Monitors for connection failures to a data source in Dataphin. Failures can be caused by network changes or incorrect credentials, which may lead to task errors. An alert is triggered when a connectivity change is detected. Table structure change: Monitors for changes in the structure of an upstream table, such as renaming, deleting, adding, or removing fields. These changes can cause downstream task errors. An alert is triggered when a table structure change is detected.
	Rule type	The rule type is derived from the selected template and can be used for description and filtering.
	Rule configuration
	Select table to check	This parameter is available only if you select Table structure change monitoring for the rule template. Select the data table to check based on the database or schema.
Business property configuration	Property information	Business property requirements depend on how the quality rule attributes are configured. For example, if the managing department attribute is set to allow multiple enum values (such as Big Data Department, Business Department, and Technology Department), you can select multiple values from a drop-down list when creating the quality rule. If the rule owner attribute is set to custom input with a length limit of 256 characters, you can enter any text up to 256 characters long. If an attribute field is set to Range, configure it as follows: Range: This is typically used for continuous numerical or date ranges. You can use the >, >=, <, and <= operators. For more information about attribute configuration, see Create and manage quality rule attributes.
Scheduling configuration	Scheduling method	You can select an existing schedule. If you have not decided on a scheduling method, you can configure it after creating the quality rule. To create a new schedule, see Create a schedule.

Click OK to save the rule configuration.

Rule configuration list

The rule configuration list displays all configured data source rules and supports viewing, editing, trial runs, runs, and deletion.

Area	Description
① Filter and search area	You can quickly search by object or rule name. You can filter by rule type, rule template, rule strength, trial run status, or effective status. Note If a business attribute is configured to be searchable or filterable and is enabled, you can use it to search or filter the rules.
② List area	Displays the object type/name, rule name/ID, trial run status, effective status, rule type, rule template, rule strength, scheduling type, and related knowledge base documents. Click the icon next to the refresh button to select which fields to display in the rule list. Effective Status: Perform a trial run before enabling a rule. Enable only the rules that pass the trial run to avoid blocking production tasks with faulty rules. After a rule is enabled, it runs automatically according to its schedule. After a rule is disabled, it no longer runs automatically, but you can still run it manually. Related knowledge base documents: Click View Details to see the knowledge base information associated with the rule. This includes the table name, check object, rule, and related documents. You can also search for, view, edit, or delete the knowledge base documents. For more information, see View a knowledge base.
③ Operations area	You can perform the following operations: View, Clone, Edit, Trial Run, Run, Scheduling Configuration, Associate Knowledge Base Documents, and Delete. View: View the details of the rule configuration. Clone: Quickly clone the rule. Edit: After you edit a rule, you must perform a new trial run. Trial Run: After a trial run, you can click the icon to view the trial run log. Run: After you run the rule, you can view the results in Check Records. Configure schedule: In the dialog box that appears, you can filter by scheduling type or quickly search by schedule name. You can also edit the schedule. Associate knowledge base documents: After you associate a rule with a knowledge base, you can view the associated knowledge in quality rules and the governance workbench. You can select a knowledge base that is not yet associated. To create a knowledge base, see Create and manage a knowledge base. Delete: Deletes the selected quality rule. This action cannot be undone.
④ Batch operations area	You can perform the following operations in batches: Trial Run, Run, Configure Schedule, Enable, Disable, Modify Business Properties, Associate Knowledge Base Documents, and Delete. Trial Run: Run trial runs for multiple rules at once. After a trial run, you can click the icon to view the trial run log. Run: Run multiple rules at once. After you run the rules, you can view the results in Check Records. Configure schedule: In the dialog box that appears, you can filter by scheduling type or search by schedule name. You can also edit schedules to configure them for multiple quality rules in a batch. You can only modify the selected rules that you have permission to edit. Enable: Enable multiple rules at once. The selected rules will run automatically based on their schedules. You can only enable the selected rules that you have permission to edit. Disable: Disable multiple rules at once. The selected rules will no longer run automatically, but you can still run them manually. You can only disable the selected rules that you have permission to edit. Modify Business Properties: You can modify business properties in batches if the corresponding attribute field is a single-choice or multiple-choice type. If the attribute is a multiple-choice type, you can append or modify the attribute values. If the attribute is a single-choice type, you can directly modify the attribute value. Associate knowledge base documents: Associate multiple quality rules with a knowledge base in a batch. After association, you can view the knowledge in the quality rules and the governance workbench. To create a knowledge base, see Create and manage a knowledge base. Delete: Delete multiple quality rules at once. This action cannot be undone. Proceed with caution. You can only delete the selected rules that you have permission to edit.

Create a schedule

Note

When you configure a schedule for a rule, you can choose from the existing schedules for the current table. Each table supports a maximum of 20 schedules.
A single rule can have a maximum of 10 schedules.
Dataphin automatically removes duplicate schedules with identical configurations.

On the Quality Rule Details page, click the Scheduling Configuration tab, and then click Create Schedule to open the Create Schedule dialog box.

In the Create Schedule dialog box, configure the parameters.

Parameter	Description
Schedule Name	Enter a custom name for the schedule. The name can be up to 64 characters long.
Scheduling type	Valid values are timed scheduling and scheduling triggered by a fixed task. Timed scheduling: Periodically runs a data quality check based on the configured schedule. This is suitable for scenarios where data is generated at a relatively fixed time. Scheduling cycle: Running a quality rule consumes computing resources. To avoid affecting production tasks, avoid running multiple quality rules concurrently. Valid scheduling cycles are Day, Week, Month, Hour, and Minute. If the system time zone (your user center time zone) is different from the scheduling time zone (configured in Management Center > System Settings > Basic Settings), the rule runs based on the system time zone. Scheduling triggered by a fixed task: Runs the quality rule before or after a specified task runs successfully. You can select tasks of the following types to trigger the rule: SQL, Offline Pipeline, Python, Shell, Virtual, Dlink, and Database SQL nodes. This type is suitable for cases where table modification tasks run on a predictable schedule. Note You can only select production environment tasks to trigger a rule. If you set the rule strength to a strong rule, a check failure may block your production tasks. Use this setting with caution based on your business needs. Supported engine types are MaxCompute. Trigger Time: Select when to trigger the quality check. Valid values are Trigger After All Tasks Run Successfully, Trigger After Each Successful Run of Each Task, and Trigger Before Each Run of Each Task. Triggering Task: The following roles can select a task node from a production project to trigger the task. You can also search by the node output name. Users with the following roles can select a task node in a production project: project administrators of Prod/Basic projects, the O&M system role for Prod projects, the developer system role for Basic projects, and custom project roles with the Project Quality Management - Quality Rule Management permission in Prod/Basic projects. Users with a custom global role that includes the Quality Rules - Management permission can select task nodes from any production project. Note If you set Trigger Time to Trigger After All Tasks Run Successfully, select triggering tasks that have the same scheduling cycle. This prevents rule execution delays and ensures timely quality check results.
Scheduling conditions	Disabled by default. If you enable this feature, the system checks if the scheduling conditions are met before the quality rule is scheduled. The rule runs only if the conditions are met; otherwise, the run is skipped. Business Date/Execution Date: You can configure dates if the scheduling type is timed scheduling (Execution Date not supported), Code-check Triggered Scheduling, or scheduling triggered by a fixed task. You can select General Calendar or Custom Calendar. For information about how to create a custom calendar, see Create and manage public calendars. If you select General Calendar, you can set conditions based on Month, Day of Week, and Date. For example: If you select Custom Calendar, you can set conditions based on Date Type and Tag. For example: Instance type: If the scheduling type is Code-check Triggered Scheduling or scheduling triggered by a fixed task, you can configure the instance type. Valid values are Scheduled Instance, backfill instance, and Manual Instance. For example: Note You must configure at least one rule. To add a rule, click +Add Rule. You can configure a maximum of 10 scheduling conditions. The relationship between scheduling conditions can be set to AND or OR.

Click OK to save the schedule configuration.

Scheduling configuration list

After a schedule is created, you can view, edit, clone, or delete it from the scheduling configuration list.

Area	Description
① Filter and search area	You can quickly search by schedule name. You can filter by timed scheduling, fixed tasks, or triggered scheduling.
② List area	Displays the Schedule Name, Scheduling Type, Last Updater, and Last Updated Time for each rule configuration.
③ Operations area	You can edit, clone, or delete a schedule. Edit: Modify the information of a configured schedule. Important This change will apply to all rules that use this schedule. Proceed with caution. Clone: Quickly copy the schedule configuration. Delete: You cannot delete a schedule that is referenced by a rule configuration.

Set up alerts

Configure different alerting methods for different rules. For example, use phone call alerts for strong rule failures and SMS alerts for weak rule failures. If a rule matches multiple alert configurations, set a policy to determine which alert takes effect.

Note

You can create a maximum of 20 alert configurations for a single monitoring object.

On the Quality Rule Details page, click the Alert Configuration tab, and then click the Create Alert Configuration button to open the Create Alert Configuration dialog box.

In the Create alert configuration dialog box, configure the parameters.

Parameter	Description
Scope	Valid values are All Rules, All strong rules, All weak rules, and Custom. Note For a single monitoring object, you can create one alert configuration for each of the following scopes: All Rules, All strong rules, and All weak rules. New rules are automatically matched to the corresponding alert based on their rule strength. To change one of these alert configurations, you must edit the existing one. For a custom scope, you can select up to 200 of the rules that are already configured for the current monitoring object.
Alert configuration name	The alert configuration name must be unique within a single monitoring object. The name can be up to 256 characters long.
Alert recipient	Configure the alert recipients and alerting methods. You must select at least one alert recipient and one alerting method. Alert recipient: Supports three types of alert recipients: custom, on-duty schedule, and quality owner. You can configure up to five custom alert recipients and up to three on-duty schedules. Alerting method: You can select from Phone, Email, SMS, DingTalk.

Click OK to save the alert configuration.

Alert configuration list

After an alert is configured, you can sort, edit, or delete it from the alert configuration list.

Area	Description
① Sorting area	Configure the policy that applies when a quality rule matches multiple alert configurations. The first matched alert configuration takes effect: If you select this policy, only the first matched alert configuration for the rule takes effect. All other configurations are ignored. You can reorder the alerts. Click Sort Rules, and then drag the icon next to the alert configuration name to reorder. You can also use the icons in the Operations column to move an alert to the top or bottom. After you finish reordering, click Finish Sorting. All alert configurations take effect: All alert configurations in the list apply to the quality rules under the current monitoring object. For example, if you configure multiple alert configurations and select this option, the system merges alerts by alerting method, alert recipient, and alert rule. If the alert recipient is the same person and the alerting method is Custom or Quality Owner, the alert messages are merged according to the merge policy. Note Alert merging is not supported for on-duty schedules.
② List area	Displays the name of the alert configuration, its effective scope, and the specific recipients and methods for each alert type. Effective Scope: Click the View icon next to a rule in the Effective Scope column to see its scope. You can only view the object name and rule name for custom alerts. If a rule is deleted, its object name cannot be viewed. Update the alert configuration in this case.
③ Operations area	You can edit or delete a configured alert. Edit: Modify the information of a configured alert. If you change the alert recipient or alerting method, notify the relevant personnel promptly to avoid missing important business alerts. Delete: After you delete an alert configuration, it will no longer apply to any matching rules. Proceed with caution.

View the quality report

Click Quality Report to view the Rule Check Overview and Rule Check Details for the current quality rule.

You can quickly filter check details by exception result, partition time, or rule/object name keyword.
In the Operations column of the rule check details list, click the icon to view the check details for the quality rule.
In the Operations column of the rule check details list, click the icon to view the execution log for the quality rule.

Permission management

Click Permission management to configure Viewable details. This setting controls which members can view check record details, quality rule details, and quality reports.

Viewable details: You can select All members or Only members with quality management permissions for the current object.
Click OK to save the permission management settings.

Next steps

After you configure the quality rule, you can view it on the data source rules page. For more information, see Manage the monitoring object list.