This topic describes how to troubleshoot and resolve latency issues for real-time synchronization tasks.
Identify the bottleneck: source or destination
For a real-time synchronization task in DataStudio, go to and click the task name to view its details. For more information, see Run and manage real-time synchronization tasks.
In the run details, check the Window Wait Time (5 min) metric. This metric shows the time the task spent waiting to read from the source or write to the destination during the last five minutes. It helps you identify the bottleneck in data synchronization. When latency occurs, the side with the higher metric value is typically the bottleneck.
Check for system exceptions
After identifying the bottleneck, go to the Logs tab. Search for keywords like "Error", "error", "Exception", "exception", or "OutOfMemory" to find exception stacks from the period of high latency. If you find an exception, use its details and consult Handle common errors to see if optimizing the task configuration resolves the issue.
A real-time synchronization task reads data from one system and writes it to another. If writing data is slower than reading it, the destination system can exert backpressure on the source system, causing it to slow down. This means a bottleneck in one system can trigger exceptions in the other. Prioritize investigating exceptions in the system identified as the bottleneck.
The following code provides a typical example of an exception stack trace:
java.lang.NullPointerException
at com.alibaba.streamx.core.util.EngineHelper.filterJobConfiguration(EngineHelper.java:31)
at com.alibaba.streamx.core.flink.trans.SinkFunctionAdaptor.open(SinkFunctionAdaptor.java:602)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
at org.apache.flink.streaming.api.operators.StreamSink.open(StreamSink.java:48)
at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:439)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:288)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:747)
at java.lang.Thread.run(Thread.java:853)Check for frequent OOM errors
In the task details, go to the Failover tab to check for frequent failovers, defined as occurring more than once every 10 minutes. If failovers are frequent, inspect the exception information for each event. If you find messages containing the OutOfMemory keyword, the task has insufficient memory and is experiencing recurring out-of-memory (OOM) issues.
To increase memory, open the task editor and increase the CU value in the Running Resources area.
Check for source data skew or partition needs
If the source is Kafka, DataHub, or LogHub and the previous steps did not reveal any exceptions or failovers, check the source system for data skew. Also, check if the read traffic for partitions or shards is reaching the synchronization rate limit.
For Kafka, DataHub, and LogHub sources, each partition or shard can be consumed by only a single parallel thread. If data is concentrated in a few partitions or shards while others are nearly empty, this data skew can create a consumption bottleneck and cause latency. This issue cannot be resolved by adjusting task settings. You must resolve the data skew in the upstream data-producing application for the Kafka, DataHub, or LogHub system. The latency will resolve once the data distribution is balanced.
In the task details dialog box, switch to the Running Information tab and review the total byte count for different reader threads. If one reader thread has a significantly higher byte count than others, data skew is likely present. However, the total byte count includes all data processed since the last processed offset. For a long-running task, this metric may not reflect recent data skew. You must also check the monitoring metrics on the source system to confirm if data skew is occurring.
If the data traffic for a single partition or shard reaches its limit, you can resolve the latency by increasing the number of partitions or shards in the source system. For example, a Kafka cluster can have a configured rate limit for partition reads, a single DataHub partition has a maximum read rate of 4 MB/s, and a single LogHub shard has a maximum read rate of 10 MB/s. If a real-time synchronization task exceeds the read speed limit for a single partition, expand the number of partitions or shards in the source system to resolve the latency.
If multiple real-time synchronization tasks consume data from the same Kafka topic, DataHub topic, or LogHub logstore, ensure that the combined read speed of all tasks does not exceed the source's limit.
Check MySQL for large transactions or frequent changes
For a real-time synchronization task with a MySQL source, if the previous steps do not reveal any exceptions or failovers, check if the source system is processing large transactions or experiencing frequent changes, such as numerous DML and DDL operations. These activities can cause the binary log to grow faster than the task can consume it, resulting in latency.
For example, updating a field across an entire table or deleting a large amount of data can cause rapid binary log growth. In the task details dialog box, switch to the Running Information tab to view the synchronization speed:
A high synchronization speed indicates that the binary log is growing quickly.
If the synchronization speed is not high, check the binary log statistics and audit logs on the MySQL server to confirm the actual growth rate.
The synchronization speed may not reflect the actual rate at which the task consumes the MySQL binary log. If a transaction or change involves databases or tables not included in the task configuration, the task filters out this data after reading it. This filtered data is not included in the synchronization speed or data volume statistics.
If you confirm that large transactions or a temporary surge in changes are causing the latency, the task will eventually catch up once it processes the backlog of changes.
Check for frequent switches in dynamic partitioning
For real-time synchronization tasks that write to MaxCompute, if you select dynamic partitioning based on field content, carefully monitor the source column that maps to the MaxCompute table's partition column. Within a single Flush Interval (default is 1 minute) configured in the Basic Configurations panel, the number of distinct values in this column should be low.
Within the flush interval, data destined for the MaxCompute table is cached in a set of queues within the real-time synchronization task. Each queue caches data for one MaxCompute write operation. The default maximum number of queues is five. If the number of distinct values from the source partition column exceeds this limit within the configured flush interval, an immediate flush of all cached data is triggered. Frequent flush operations severely degrade write performance.
You must confirm if frequent flushes are triggered because the MaxCompute table partition cache queues are exhausted. In the task details dialog box, switch to the Logs tab and search for the message uploader map size has reached uploaderMapMaximumSize.
Increase concurrency or enable distributed execution
If the previous steps show that latency is due to increased source traffic rather than exceptions, you can mitigate it by increasing the task's concurrency.
When you increase concurrency, you must also increase the task's memory. As a general rule, add 1 GB of memory for every four additional parallel threads.
Configure the task concurrency and memory as follows:
For single-table-to-single-table ETL real-time synchronization tasks created in DataStudio, click Basic Configurations on the right to configure task concurrency and memory. In the Basic Settings panel, you can configure parameters such as Synchronization Method, the distributed execution mode switch, resource group, CUs, and number of parallel threads. Expand Advanced Settings to configure the flush interval (default: 60000 ms) and MaxCompute Channel Resources. For example, select Streaming Tunnel and set the slot number.
For other DataStudio tasks, like database migrations to DataHub, configure the number of parallel threads in the Configure Resource step and memory in the Basic Configurations panel.
For synchronization solution tasks, you can configure the number of parallel threads and memory in the Configure Resource step.
If distributed execution mode is disabled, set the number of parallel threads to 32 or fewer. Setting the count above 20 may cause latency due to single-machine resource bottlenecks. For specific channels, you can enable distributed execution mode to improve performance. The channels that support distributed execution mode are listed in the following table.
Task type | Source | Destination |
DataStudio ETL task | Kafka | MaxCompute |
DataStudio ETL task | Kafka | Hologres |