To ensure that tasks run properly, after you develop tasks in Data Studio and deploy them to the production environment, you can go to Operation Center to manage batch synchronization tasks, monitor task running status, change resource groups for synchronization, and view runtime logs. This topic describes common operations for batch synchronization tasks.
Overview
-
O&M for batch synchronization tasks is the same as for other scheduled tasks. This topic uses batch synchronization as an example. For more information about O&M for scheduled tasks, see Perform basic O&M operations on scheduled tasks.
-
Workspaces in standard mode support data source environment isolation. A data source with the same name can have two separate configurations for the development and production environments. You can set up two different databases or instances. This isolates data operations during testing and production scheduling, which ensures the security of your production data.
Access the Operation Center
Log on to the DataWorks console, switch to the target region, and then in the left-side navigation pane, select . Select your workspace from the drop-down list and click Operation Center.
Schedule and manage tasks
Batch synchronization relies on the powerful scheduling capabilities of DataWorks to automatically run tasks on a recurring basis. You can use DataWorks scheduling parameters to write both incremental and full data to specific partitions of a destination table. O&M for batch synchronization tasks is the same as for other scheduled tasks. They can be run manually or triggered automatically on a schedule.
|
Actions |
Description |
|
Run a batch synchronization task |
After you deploy a task, go to the page to view the deployed task. The scheduling system automatically runs the task based on its schedule settings. You can also trigger the task manually.
|
|
Pause scheduling of a batch synchronization task |
On the Auto Triggered Task page, you can pause (freeze) a single task to stop its scheduled runs for a period of time and block the execution of its downstream nodes. Note Running a scheduled task generates instances. If a specific instance and its downstream instances do not need to run, you can freeze the instance. |
|
Resume scheduling of a batch synchronization task |
On the Auto Triggered Task page, you can resume (unfreeze) a frozen task so that it resumes normal scheduling. Note Running a frozen scheduled task generates frozen instances. If the frozen instances and their downstream instances need to run, you can unfreeze the instances. |
Synchronize historical data
By using DataWorks scheduling parameters together with the backfill data feature, you can batch synchronize historical data to a specified table or partition in the destination database or data warehouse based on a single task configuration. If your task is configured for incremental synchronization and you need to synchronize historical data to the corresponding partitions of the destination table, you can use the backfill data feature. The scheduling parameters are automatically replaced with specific values based on the business date configured for the backfill operation. For more information about the backfill data feature, see Backfill data.
Monitor task status
You can go to to configure monitoring rules to monitor the running status of tasks, including completed, incomplete, error, and timeout statuses. For more information, see Configure monitoring rules.
Resource group operations
-
Resource group monitoring: You can monitor the usage of resource groups used by tasks in Operation Center. For more information, see Monitor resource groups.
-
Resource group switching: You can switch the resource group used by tasks in the following ways.
NoteBefore you change the resource group, make sure that the target resource group is connected to the data source network to avoid task execution failures caused by network connectivity issues.
Environment
Supported switching operations
Entry point
Switch the production environment resource group
Batch switching
Go to the page to configure the settings.
Select the tasks for which you want to change the resource group, and click Change Resource Group for Data Integration in the bottom toolbar to change the resource group in a batch.
Switch the development environment resource group
NoteAfter you switch the resource group in the development environment, you must re-submit and deploy the task to the production environment.
-
Single task switching
-
Batch switching
Go to the DataStudio page to configure the settings.
-
Single task switching
Go to the editing page of the target node and modify the resource group in the resource section. In the Resource section, set Resource Group to dwGroup (10 CUs, subscription), and set Resource Usage (CU) to
1 CU. Click Connectivity Test and confirm that both the Source and Destination MaxCompute (ODPS) show Connected. -
Batch switching
Click the
icon, select the tasks for which you want to change the resource group, and click Change Resource Group for Data Integration in the bottom toolbar to change the resource group in a batch.
-
Monitor data quality of tables
Some destination data sources in Data Integration support data quality monitoring for output tables. You can go to the data quality rules page to configure data quality rules for the destination tables to which data is written. For tables with data quality monitoring rules configured, data quality checks are triggered after the associated scheduled node finishes running (executing the node code logic). The DataWorks platform determines whether the task should exit due to a quality rule check failure based on the rule type (strong rule or weak rule) and the check result. A failed strong rule blocks downstream node execution to prevent the spread of dirty data. For information about whether the destination data source in Data Integration supports data quality monitoring and how to use it, see Data quality monitoring.
If data quality rules are configured for the output tables of batch synchronization tasks, pay attention to the network connectivity between the resource group used by the task and the destination data source.
View runtime logs of batch tasks
Go to Operation Center. After scheduled instances, backfill instances, or test instances finish running, you can view the Operational Logs of the task on the instance DAG panel. For more information, see View instance details.
To learn about the parameters in the logs during execution, see Log parameters.
View the batch synchronization dashboard
You can also go to to view runtime metrics of batch synchronization tasks, including the running status distribution, data synchronization progress, synchronized data volume statistics, and task execution details (which can be filtered by source and destination data sources, and whether the task uses the public network). For more information, see View the batch synchronization dashboard.
Use Logview to view task running information
Data Integration Logview collects, analyzes, and visualizes detailed synchronization metrics such as synchronization rates and logs at a finer granularity during data synchronization tasks.
-
In Operation Center, find the running instance of the batch task and click Perform Diagnosis. On the Instance View page, find the target task in the instance list and click Run Diagnosis in the Actions column.
-
On the Run diagnosis page, click the tab to view error log analysis results from large language models to assist with troubleshooting.
The intelligent diagnosis section provides analysis from three large language models: Qwen, DeepSeek, and DW Knowledge Base, showing Key Error Logs, Error Analysis, Suggested Solutions, and Quick Actions (including Edit Code, Rerun, and Go to Data Map). A status progress indicator at the top of the page shows the pass/fail status of each stage: Upstream Dependency, Scheduled Check, Scheduling Resource, and Task Execution.
Parameter
Description
Logs
On the Logs page, you can view detailed logs of Data Integration synchronization tasks.
Progress
On the Progress page, you can view detailed progress information of Data Integration synchronization tasks, including Synchronized Data Records, Synchronized Bytes, Synchronization Rate for Synchronized Data Records, and Synchronization Rate for Synchronized Bytes.
You can also perform the following operations on this page:
-
You can use the time filter to filter synchronization information for a specific time range.
NoteThe time filter supports viewing detailed information for up to 15 days.
-
On the right side of the process list, click the
icon to display more columns. In the process list area, you can use the column settings dialog to select the monitoring metric columns to display. Available columns include Subtask ID, Start Time, Synchronized Records, Average Synchronization Rate (records), Average Synchronization Rate (bytes), Execution Machine, Average GC Duration, GC Count, Status, End Time, Duration, Reader Wait Time, Writer Wait Time, and Dirty Data Count. -
In the process list, you can click a specific item to view the trend chart of a metric.
Instance Overview
If the instance is a scheduled instance, you can also go to the Instance Overview page to view comparison details across different cycles for the instance.
-
You can view the status of the task and the corresponding Instance ID in the Task List. Click an Instance ID to view the task details.
-
You can also use the bar chart to compare the differences in Sync Rate, Synchronized Data Records, Waiting time, and Synchronization Duration across different instances.
-
Next steps: Data quality issues and task optimization for batch synchronization
Full synchronization progress of a database-level batch synchronization task stays at 0%
If the full synchronization progress of a database-level batch synchronization task continuously shows 0%, troubleshoot the following causes one by one.
-
Check whether the task is in the WAIT state
Check the instance status in Operation Center. The WAIT state indicates that the task is waiting for upstream dependencies to complete or for scheduling resource allocation, and the full synchronization progress does not advance. After the upstream dependencies are met and resources are ready, the task starts running automatically.
-
Check whether the resource group has sufficient CUs
Insufficient CUs in the resource group cause tasks to queue. Check the CU usage on the resource group monitoring page. If the usage is consistently close to 100%, scale up the resource group or reduce the number of concurrent tasks.
-
Check whether network connectivity is normal
If the network between the resource group and the data source is not connected, the synchronization task cannot read or write data. Run a connectivity test on the data source configuration page to confirm that both the source and destination are connected.
-
Check whether subtasks have been generated and are running
A database-level batch synchronization task consists of multiple subtasks. Check the subtask list on the Progress page in Logview to confirm that subtasks have been generated and are in the running state. If no subtasks have been generated, verify that the table selection rules in the task configuration correctly match the source tables.