Analyze batch synchronization logs

Area

Parameter

Description

Submit instance (Area ①)

SUBMIT: The scheduling system has submitted the synchronization task to a resource group for Data Integration. This indicates that the system has processed the task definition.

The scheduling system dispatches the task to a resource group for execution. Area ① shows the Data Integration resource group that the task is using. The log output varies based on the resource group type:

• If the task runs on the default resource group, the log displays the following message:

  running in Pipeline[basecommon_ group_xxxxxxxxx]

• If the task runs on an exclusive resource group for Data Integration, the log displays the following message:

  running in Pipeline[basecommon_S_res_group_xxx]

• If the task runs on a serverless resource group, the log displays the following message:

  running in Pipeline[basecommon_Serverless_res_group_xxx]

Request resources (Area ②)

WAIT: The synchronization task is waiting for Data Integration execution resources to become available.

If a task remains in the WAIT state for an extended period, other tasks may be occupying the resource group's resources. You can resolve this issue in the following ways:

• Wait for the occupying tasks to complete and release resources, and then start your task. To identify the resource-intensive tasks, see Scenarios and solutions for slow data synchronization.

• Identify the tasks occupying resources and coordinate with their owners to reduce concurrency.

• Reduce the concurrency of the current synchronization task and resubmit the task.

• Scale out the resource group. For more information, see Scale-out and scale-in operations.

Start synchronization (Area ③)

RUN: The synchronization task is in progress.

A batch synchronization task has four execution stages:

1. Pre-SQL execution

   Based on your configuration, the system sends a pre-SQL statement to the database. Not all tasks include this stage.

   • For example, for a MySQL writer, if you have configured a PreSQL statement to run before the data synchronization task, that SQL statement is executed in this stage.

   • For example, for a MySQL reader, if you have configured a querySql statement or a where clause for data filtering, those SQL statements are executed in this stage.

   • For example, when you write data to MaxCompute, you can enable Delete Existing Data Before Writing.

   Note

   We recommend using indexed fields in filter conditions to prevent long-running SQL queries, which can increase overall synchronization time or cause database timeouts that interrupt the task.

2. Task splitting

   In this stage, the system splits the source data into multiple sub-tasks for concurrent, batched reading. The splitting rules are as follows:

   • Relational databases: The system splits data into multiple tasks based on the specified shard key (splitPk). These tasks are then read in concurrent batches. If no shard key is set, the task uses a single channel for synchronization.

   • LogHub, DataHub, and MongoDB: Data is split by the number of shards. The maximum task concurrency cannot exceed the number of shards.

   • Semi-structured storage: Data is split by the number of files or data volume. For example, for an OSS task, the maximum concurrency cannot exceed the number of files.

3. Data synchronization

   In this stage, the system synchronizes the split tasks in batches according to your configured concurrency. For a relational database, the shard key is used to generate multiple data retrieval SQL statements that request data from the database in parallel. For more information, see Relationship between batch synchronization concurrency and rate limiting.

   Note

   • The actual concurrency during execution may differ from the value you set.

   • An improperly configured shard key can lead to long-running SQL queries, which can increase overall synchronization time or cause database timeouts that interrupt the task.

   • High database load can also slow down task execution.

4. Post-SQL execution

   Based on your configuration, the system sends a post-SQL statement to the database. Not all tasks include this stage.

   • For example, for a MySQL writer, if you have configured a PostSQL statement to run after data synchronization, that SQL statement is executed in this stage.

   • The execution time of the PostSQL statement contributes to the total task execution time.

Execution completed (Area ④)

There are two completion statuses:

• FAIL: The synchronization task failed.

• SUCCESS: The synchronization task was successful.

• If the task fails, the log displays a key error message. You can click the link in Area ⑤ to view the detailed execution process for each stage.

• If the task succeeds, the log shows a summary, including the total number of synchronized records and the average synchronization speed.

Detailed log link (Area ⑤)

A link to the detailed log.

Click the link to view detailed logs for each execution stage.