DocsICP FilingConsole
Documentation
Home Page DataWorks User Guide Data Governance Data quality Configure monitoring rules Configuration example: MaxCompute

Configuration example: MaxCompute

Updated at:
Copy as MD
Product
My Favorites

Monitoring rules are the core of Data Quality (DQC). You can use Data Quality to monitor data in MaxCompute, E-MapReduce (EMR), Hologres, AnalyticDB for PostgreSQL, and AnalyticDB for MySQL. This topic explains how to configure a monitor for MaxCompute.

Procedure

Step 1: Add a MaxCompute data source

    1. Log on to the DataWorks console. In the target region, click More > Management Center in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Management Center.

    2. On the Workspace Management page, click Data Sources in the left-side navigation pane to open the data source page.

  2. Click Add Connection to add a MaxCompute data source. For more information, see Associate a MaxCompute compute resource.

  3. Click the icon icon in the upper-left corner of the current page, and select All Products > Data Development and O&M > DataStudio.

  4. In the left-side navigation pane of the DataStudio page, click computing resource to bind the MaxCompute computing resource you added.

    Note

    In this example, the data in the test table is from a MaxCompute public dataset. When you configure monitoring for your actual business data, you must create a monitor and monitoring rules based on your specific needs.

Step 2: Go to the table quality details page

  1. Click the icon icon in the upper-left corner of the current page, and select All Products > Data Governance > Data Quality.

  2. In the left navigation bar, click Rule Setting > Configure by Table.

  3. In the left Connection list, select a database, find the table for which you want to configure monitoring rules, and click the table name or Configure Monitoring Rule in the Actions column to go to the Table Quality Details page.

    Note

    You can also search for a table by entering its name. The search supports fuzzy matching based on the first few letters of the table name.

Step 3: Create a monitor

  1. On the Rule Management tab of the Table Quality Details page, click the image icon next to Monitor Perspective, and the Create Monitor page opens.

  2. Configure the parameters for the monitor.

    Note

    You can create a monitor based on the data validation requirements for your table. For more information about how to create a monitor, see Configure rules: By table.

    On the Create Monitor page, set Monitor Name (for example, mc table row count greater than 2000 monitor), select the target table (for example, mc_test_tb) for Monitored Object, select Set range by partition for Data Range and enter ds=${yyyymmdd-1} for the partition expression, enter a description of the monitor in Description, select Manual for Trigger Method, select the desired resource group for Resource Group, set Issue Handling Policy to Alert, click Batch Select in the Select Monitoring Rules section to add rules, select Subscribe to Quality Alerts (Email), and then click Save.

  3. Click Save.

Step 4: Configure a monitoring rule

  1. On the Rule Management tab of the Table Quality Details page, click Create Rule, and the rule configuration page opens.

  2. This example uses a built-in template to create a monitoring rule.

    Note

    You can create rules based on the actual requirements for your table. For more information about how to create rules, see Configure rules: By table.

    In the left-side System Templates list, expand the Table Row Count category and select the Table Row Count, Fixed Value template. In the right-side form, set Rule Template to Table Row Count, Fixed Value, set Rule Scope to mc_test_tb, select Manual for Comparison Method, set Monitoring Threshold to normal threshold <= 2000 and red threshold > 2000, enable Status, and select Weak Rule for Severity.

  3. Add the monitoring rule to the monitor you created in Step 3.

    Select Add to Monitor, select the monitor created in Step 3 from the drop-down list, optionally select Add me as an alert subscriber for this monitor, and then click OK.

  4. Click Confirm.

Step 5: Test rule execution

  1. On the Table Quality Details page, on the Monitor tab, find the monitor that you created and click Test in the Actions column.

  2. In the Tests dialog box, confirm the Data Scope and Scheduled Time parameters and click Tests.

    The dialog box also displays the Monitor name and the number of enabled rules.

  3. When Started is displayed, you can click View Details to view the detailed results of the test run.

    In this example, the MaxCompute test table contains more than 2,000 rows. As a result, the quality check identifies an anomaly, which is indicated in red.

    In the Quality Collection and Comparison Status Results section, the check status is Anomaly, the issue handling method is Alert, and the check result shows that the sampled value is 65000000 and the check value is 65000000, far exceeding the configured threshold of 2000.

  4. After you confirm that the test rule is triggered successfully, you can modify the monitor and change the Trigger Method to Triggered by Node Scheduling in Production Environment. You can then associate the monitor with a specified scheduling node in the DataWorks Operation Center. After the node finishes running, the quality rules under this monitor are automatically triggered.

Step 6: Configure an alert subscription

  1. On the Table Quality Details page, on the Monitor tab, find the monitor that you created and click More > Alert Subscription in the Actions column.

  2. After you configure the Notification Method and Authorized object, click Save.

References

Previous：Configure noise reduction for data quality rulesNext：Resolve data quality issues