Configure data verification

Usage notes

• DTS does not support incremental data verification for database instances in a classic network.

• DTS does not support data verification for instances with the multi-table merging feature enabled.

• DTS does not support data verification for instances whose Status is Completed.

• DTS does not verify a table if it has over 10,000 rows and lacks a primary key or unique index.

• If the data has neither a primary key constraint nor a unique constraint, DTS does not perform incremental data verification.

• If you restart a synchronization or migration instance, for example, by pausing and then starting it, the full verification task in the instance restarts.

• DTS does not verify an empty destination table.

• Before you perform a verification task, assess the performance of the source and destination databases. Perform data verification during off-peak hours, for example, when the CPU load of the source and destination databases is below 30%. During data verification, DTS consumes read resources from both the source and destination databases, which may increase the database load. If the database has poor performance, low specifications, or a heavy workload, the increased load can make its services unavailable.

• When you configure a data verification task in a data synchronization or migration instance, your selections for Synchronization Types or Migration Types determine the available options for Data Verification Mode.

  For example, if you select only Incremental Data Migration for Migration Types, Incremental Data Verification is the only available option for Data Verification Mode.

• An incremental data verification task verifies only data changed during an incremental task. It does not verify data manually changed in the destination database.

• DTS does not support incremental verification for data verification tasks configured from the data verification task list.

• When you verify a small amount of data using the Full field validation by row sampling method, the specified sampling percentage has no effect. The value defaults to 100%.

• If you select the Full field validation by row sampling method for data verification on a MongoDB instance, the sampling percentage is fixed at 100% and cannot be modified.

• If you select Full field validation by row sampling for Full Data Verification, do not use the ETL feature at the same time. Otherwise, data modified by the ETL feature is flagged as inconsistent.

• DTS does not support data verification for materialized views.

• During a full verification task, if a single table contains more than 100,000 rows of inconsistent data, DTS stops the verification of that table. Reduce the amount of inconsistent data and then verify the table again.

• If an object contains hidden columns, it can affect the schema verification result.

• You are charged for incremental validation tasks even when they are paused.

Supported regions

For more information about supported regions, see list of supported regions.

Supported links

Configure a data verification task

1. Go to the data verification task list page. Choose one of the following methods.

   DTS console (quick)

   1. Go to the Data Verification page.

   2. Optional: In the upper-left corner of the page, select the region of the verification instance.

   DTS console (standard)

   1. Log on to the Data Transmission Service (DTS) console.

   2. In the left-side navigation pane, click Data Verification.

   3. Optional: In the upper-left corner of the page, select the region of the verification instance.

   DMS console

   Note

   The actual steps may vary based on the mode and layout of the DMS console. For more information, see Simplified mode console and Customize the layout and style of the DMS interface.

   1. Log on to the Data Management Service (DMS) console.

   2. In the top menu bar, choose Data + AI > Data Transmission (DTS) > Data Verification.

   3. Optional: To the right of Verification Task, select the region of the verification instance.

2. Navigate to the configuration page for the data verification task by using one of the following methods:

   • If you have purchased a data verification task (purchase before configuration), find the task and click Configure Task in the Actions column.

     To learn how to purchase a data verification task, see Purchase a data verification task.

   • If you have not purchased a data verification task (configure before purchase), click Create Task.

3. Configure the source and destination databases. For details, see Overview of data synchronization solutions or Overview of data migration solutions.

   Note

   • You cannot change the Instance Region for pre-purchased tasks.

   • If the source database is a MongoDB sharded cluster, you do not need to provide a Shard account or Shard password.

4. After you complete the configuration, click Test Connectivity and Proceed at the bottom of the page.

   Note

   • Ensure that the IP address segment of the DTS service is automatically or manually added to the security settings of the source and destination databases to allow access from DTS servers. For more information, see Add DTS server IP addresses to a whitelist.

   • If the source or destination database is a self-managed database (the Access Method is not Alibaba Cloud Instance), you must also click Test Connectivity in the CIDR Blocks of DTS Servers dialog box that appears.

5. Set Data Verification Mode.

   Select a data verification method. If you select Full Data Verification, you must also configure the parameters described in the following table.

   Note

   • Full Data Verification: Verifies the full data of the specified objects.

   • Schema Verification: Verifies the database and table schemas of the specified objects.

   • DTS does not support incremental verification for data verification tasks configured from the data verification task list.

   Parameter	Description
   Full Data Verification	Full field validation by row sampling: Specify a sampling percentage to verify all fields of the sampled rows. The value must be an integer from 10 to 100. Verify based on the number of table rows: Verifies the row count of the full data task. The actual data content is not verified. Note The Verify based on the number of table rows mode is free of charge. The Full field validation by row sampling mode is charged based on the amount of data that is actually verified.
   Full Data Verification Time Rule	Currently, only Start Now is supported.
   Timeout Settings for Full Data Verification	No: The full data verification task is not forcibly terminated on timeout. Yes: Specify a timeout period for the full data verification task. The timer starts after the task is initiated. If the task does not complete within the specified period, it is forcibly terminated. The value must be an integer from 1 to 72.
   Full calibration reference	Default: Uses the union of the source and destination databases as the benchmark to verify data consistency. Source Database: Uses the source database as the benchmark to verify data consistency. Extra data in the destination database is not verified. Destination Database: Uses the destination database as the benchmark to verify data consistency. Extra data in the source database is not verified.
   Maximum number of rows of data read per second by full verification (RPS)	Full data verification consumes read resources from the database. To reduce the load on the database, you can set rate limits for the full data verification task, such as the maximum rows per second (RPS) and the maximum read throughput in megabytes per second (MBps). Note A value of 0 indicates no limit. If both Maximum number of rows of data read per second by full verification (RPS) and Maximum amount of data read per second by full verification (MBps) are set to 0, no rate limiting is applied.
   Maximum amount of data read per second by full verification (MBps)

6. Set Verification Objects.

   1. Select the objects to verify in the source database.

      In the Source Objects box, select the objects to verify and click to move them to the Selected Objects box.

      Note

      You can set the Table Name and Column Name for destination tables only when you set the verification object granularity to tables.

   2. Set the objects to verify in the destination database.

      1. In the Selected Objects box, right-click the schema that you moved.

      2. In the Edit Schema dialog box, set the Schema Name for the destination database and click OK.

      3. In the Selected Objects box, right-click the table that you moved.

      4. In the Edit Table dialog box, set the Table Name for the destination table.

      5. Optional: In the Column section, set the destination column name in the Column Name column.

         Note

         You can also clear the Sync All Columns checkbox and select the specific columns to verify.

      6. After you complete the settings, click OK.

   3. Optional: Configure verification alerts.

      If you select Full Data Verification for the Data Verification Mode parameter, you can also configure Full Data Verification Alert.

      Note

      You can also set or modify data verification alerts after the DTS instance is running.

      Parameter	Description
      Full Data Verification Alert	No: No alert is configured. Yes: Select and configure one of the following alert rules: Triggers an alert if the full data verification task fails. Triggers an alert if the number of inconsistent records from the full data verification task is greater than or equal to a specified threshold.

      Note

      If you enable verification alerts, you must also enter the mobile phone number of the Alert Contact. When the data verification task triggers an alert, DTS sends an SMS message to the alert contact.

7. Click Next: Advanced Settings.

   Configuration	Description
   Dedicated Cluster for Task Scheduling	By default, DTS schedules the task on a shared cluster, and no selection is required. You can purchase a dedicated cluster to run DTS tasks. For more information, see What is a DTS dedicated cluster?.
   Retry Time for Failed Connections	If the connection to the source or destination database fails after the synchronization task starts, DTS reports an error and immediately begins to retry the connection. The default retry duration is 720 minutes. You can customize the retry time to a value from 10 to 1,440 minutes. We recommend a duration of 30 minutes or more. If the connection is restored within this period, the task resumes automatically. Otherwise, the task fails. Note If multiple DTS instances (e.g., Instance A and B) share a source or destination, DTS uses the shortest configured retry duration (e.g., 30 minutes for A, 60 for B, so 30 minutes is used) for all instances. DTS charges for task runtime during connection retries. Set a custom duration based on your business needs, or release the DTS instance promptly after you release the source/destination instances.
   Retry Time for Other Issues	If a non-connection issue (e.g., a DDL or DML execution error) occurs, DTS reports an error and immediately retries the operation. The default retry duration is 10 minutes. You can also customize the retry time to a value from 1 to 1,440 minutes. We recommend a duration of 10 minutes or more. If the related operations succeed within the set retry time, the synchronization task automatically resumes. Otherwise, the task fails. Important The value of Retry Time for Other Issues must be less than that of Retry Time for Failed Connections.
   Environment Tag	This parameter is optional. You can select an environment tag to identify the instance based on your business requirements.

8. Save the task and run a precheck.

   • To view the parameters for configuring this instance when you call the API operation, move the pointer over the Next: Save Task Settings and Precheck button and click Preview OpenAPI parameters in the bubble that appears.

   • If you do not need to view or have finished viewing the API parameters, click Next: Save Task Settings and Precheck at the bottom of the page.

   Note

   • DTS runs a precheck before the data verification task starts. The task starts only after it passes this precheck.

   • If the precheck fails, click View Details next to the failed check item, fix the issue based on the prompt, and then run the precheck again.

   • If a warning is reported during the precheck:

     • For check items that cannot be ignored, click View Details next to the failed item, fix the issue based on the prompt, and then run the precheck again.

     • For check items that can be ignored, you can click Confirm Alert Details, Ignore, OK, and Precheck Again to skip the alert item and run the precheck again. If you choose to ignore a warning, it may cause issues such as data inconsistency and pose risks to your business.

9. View the progress of the data verification task.

   • For a pre-purchased task: When the Success Rate is 100%, click Back.

   • For a task configured before purchase: When the Success Rate is 100%, click Next: Purchase Instance, read and select Data Transmission Service (Pay-as-you-go) Service Terms, click Buy and Start, and then click OK.

Configure data verification

1. Go to the Data Verification page.

   • For a new DTS instance: To go to this page, see the relevant configuration topics in Overview of data synchronization solutions or Overview of data migration solutions.

   • For an existing DTS instance:

     1. Go to the list of synchronization or migration tasks.

     2. Click the ID of the target instance.

     3. On the Basic Information or Task Management page, click Create Verification Task.

2. Configure the Data Verification Mode.

   Note

   • Full Data Verification: Verifies the data in the full data task. If the instance does not include an incremental task, verification starts after the instance is created and the full data is written. If the instance includes an incremental task, verification starts after the instance is created and replication latency drops to zero for the first time.

   • Incremental Data Verification: Verifies the data in the incremental task. Verification starts after the instance is created and replication latency drops to zero for the first time.

   • Schema Verification: Verifies the schemas of the selected objects. If the instance does not include an incremental task, verification starts after the instance is created and the table schema and full data are written. If the instance includes an incremental task, verification starts after the instance is created and replication latency drops to zero for the first time.

   Full data verification

   If you select Full Data Verification, you must also configure the parameters described in the following table.

   Parameter	Description
   Full Data Verification	Full field validation by row sampling: Specify a sampling percentage to verify all fields of the sampled rows. The value must be an integer from 10 to 100. Verify based on the number of table rows: Verifies the row count of the full data task. The actual data content is not verified. Note The Verify based on the number of table rows mode is free of charge. The Full field validation by row sampling mode is charged based on the amount of data that is actually verified.
   Full Data Verification Time Rule	Currently, only Start Now is supported.
   Timeout Settings for Full Data Verification	No: The full data verification task is not forcibly terminated on timeout. Yes: Specify a timeout period for the full data verification task. The timer starts after the task is initiated. If the task does not complete within the specified period, it is forcibly terminated. The value must be an integer from 1 to 72.
   Full calibration reference	Default: Uses the union of the source and destination databases as the benchmark to verify data consistency. Source Database: Uses the source database as the benchmark to verify data consistency. Extra data in the destination database is not verified. Destination Database: Uses the destination database as the benchmark to verify data consistency. Extra data in the source database is not verified.
   Maximum number of rows of data read per second by full verification (RPS)	Full data verification consumes read resources from the database. To reduce the load on the database, you can set rate limits for the full data verification task, such as the maximum rows per second (RPS) and the maximum read throughput in megabytes per second (MBps). Note A value of 0 indicates no limit. If both Maximum number of rows of data read per second by full verification (RPS) and Maximum amount of data read per second by full verification (MBps) are set to 0, no rate limiting is applied.
   Maximum amount of data read per second by full verification (MBps)

   Incremental data verification

   If you select Incremental Data Verification, you must also configure the parameters described in the following table.

   Parameter	Description
   Incremental Verification Benchmark	You can select the DML operations to verify.

3. Specify Verification Objects.

   In the Selected Objects box, select the objects to exclude from verification and then click the icon to remove them.

   Note

   By default, DTS adds the objects selected for synchronization or migration to the Selected Objects box.

4. Configure verification alerts.

   Configure the following parameters as needed.

   Note

   You can also set or modify data verification alerts after the DTS instance is running.

   Parameter	Description
   Full Data Verification Alert	No: No alert is configured. Yes: Select and configure one of the following alert rules: Triggers an alert if the full data verification task fails. Triggers an alert if the number of inconsistent records from the full data verification task is greater than or equal to a specified threshold.
   Incremental Data Verification Alert	No: No alert is configured. Yes: Select and configure one of the following alert rules: Triggers an alert if the incremental data verification task fails. Triggers an alert if the number of inconsistent records meets or exceeds a specified threshold across a defined number of consecutive statistical periods. Triggers an alert if data latency meets or exceeds a specified threshold across a defined number of consecutive statistical periods.

   Note

   If you configure a verification alert, you must also enter the mobile phone number of the Alert Contact. When the verification task triggers an alert, DTS sends an SMS message to the alert contact.

5. Follow the on-screen prompts and refer to the relevant topics to complete the configuration.

Modify a verification alert

1. Go to the data verification task list page using one of the following three methods.

   Note

   If you configured the data verification task in a Data Transmission Service (DTS) synchronization or migration instance, you can also go to the corresponding synchronization or migration task list page.

   DTS console (quick)

   1. Go to the Data Verification page.

   2. Optional: In the upper-left corner of the page, select the region of the verification instance.

   DTS console (standard)

   1. Log on to the Data Transmission Service (DTS) console.

   2. In the left-side navigation pane, click Data Verification.

   3. Optional: In the upper-left corner of the page, select the region of the verification instance.

   DMS console

   Note

   The actual steps may vary based on the mode and layout of the DMS console. For more information, see Simplified mode console and Customize the layout and style of the DMS interface.

   1. Log on to the Data Management Service (DMS) console.

   2. In the top menu bar, choose Data + AI > Data Transmission (DTS) > Data Verification.

   3. Optional: To the right of Verification Task, select the region of the verification instance.

2. In the Actions column for the target instance, choose > Alert Settings.

   Note

   Alternatively, click the ID of the target instance, and then click Alert Settings in the left-side navigation pane.

3. In the Actions column for the target metric, click Edit.

   • full data verification

     • Full Data Verification Status

     • Full Data Verification Inconsistency

   • incremental data verification

     • Incremental Data Verification Status

     • Incremental Data Verification Latency

     • Incremental Data Verification Inconsistency

4. After you modify the settings, click OK.

Related APIs

FAQ

• Are database, table, and column names modified by the name mapping feature flagged as inconsistent data?

  Data verification supports the name mapping feature for databases and tables. If you use this feature to modify database, table, and column names, DTS flags only the modified column names as inconsistent, not the database or table names.

• How does schema verification handle tables with hidden columns?

  Schema verification checks the position of columns. If a table being verified contains a hidden column, this can affect the positions of other columns. If the position of a column in the source database is different from its position in the destination database, DTS flags a schema inconsistency.

• Why can't I select a verification mode in a DTS instance?

  The instance may not support your selected data verification mode, or you may not have selected the corresponding Configure Objects or Migration Types in the Synchronization Types step.

• Is data filtered by the Filter Conditions feature flagged as inconsistent?

  • Full data verification: No. Full data verification supports the Filter Conditions feature.

  • Incremental data verification: No. Incremental data verification supports the Filter Conditions feature. However, incremental tasks do not support the Filter Conditions feature if the source Database Type is MongoDB.