Single-table real-time synchronization: O&M and tuning

Start and stop

• After you start a task, first verify that schema preparation is complete, then observe whether full initialization has started and is progressing, and finally confirm that real-time incremental synchronization is continuously reading and writing data. For message source tasks, also monitor the consumer offset and message backlog. For database log-based tasks, monitor Binlog, WAL, or log retention time.

• Before you stop a task, confirm whether the source log or message retention period can cover the downtime window. If the downtime exceeds the retention period, the task may not be able to resume from the original offset. In such cases, you typically need to reset the offset, skip the historical backlog, or reinitialize. Before performing these operations, evaluate the risks of duplicate writes and data loss.

Modify configurations

Common modifications for single-table real-time tasks include adjusting column mappings, primary keys, write modes, partition rules, concurrency, resources, offsets, and dirty data handling policies.

Alert configuration

For single-table real-time tasks, monitor at least task status, business latency, and Failover. Depending on the channel capabilities and task configuration, also add alerts for resource utilization, write errors, DDL notifications, and message backlog.

For message source tasks, also monitor the backlog of the source topic, shard, Logstore, or consumer group. For database log-based tasks, monitor Binlog, WAL, archive log, or replication slot retention. Configuring only task failure alerts cannot detect issues where the task is still running but latency is continuously increasing, writes are slowing down, or offsets have not advanced for a long time.

Task produces no data output

When the task shows as running but no new data appears at the destination, confirm whether the source actually has new data, then determine whether the reader is not consuming data, the pipeline is experiencing processing errors, or the writer is not committing data.

Troubleshooting sequence:

1. Check whether the source table, topic, Logstore, or log offset has new data.

2. Check whether the task is in the full initialization, real-time incremental synchronization, or Failover state.

3. Check whether read metrics, write metrics, business latency, and Checkpoint are progressing normally.

4. Check whether the destination table, destination partition, primary key, and write mode match the configuration.

5. Check whether dirty data, column mapping failures, or destination permission issues exist.

If the DML/DDL statistics in the runtime details show no data for a short period but the logs and destination writes are normal, also check for metric collection latency or collection anomalies to avoid misinterpreting a monitoring display issue as a synchronization interruption.

Slow or failed full initialization

When full initialization is slow, first identify whether the bottleneck is in source reads, network, resource group, destination writes, or uneven data splitting.

During the full initialization phase, do not rely solely on average throughput. A few large shards, wide tables, large columns, or hot partitions may determine the overall completion time.

Increasing real-time latency

When real-time latency increases, check whether the latency is continuously decreasing. If the task has just completed full initialization, the real-time pipeline may be replaying incremental data accumulated during the full synchronization phase. If the latency continues to grow, identify whether the bottleneck is in the reader, writer, resources, network, or an external system.

We recommend that you troubleshoot in the following order:

1. Check the window wait time in the task runtime details to determine whether the bottleneck is primarily in the reader or writer.

2. Check the task logs during the high-latency period and search for keywords such as Error, Exception, and OutOfMemory.

3. Check the Failover records to determine whether frequent Failovers, OOM errors, or external service exceptions have occurred.

4. Check the runtime metrics and events, focusing on throughput, backpressure, Checkpoint, JVM memory, GC, DDL events, and dirty data.

5. Then continue to identify the cause by examining the source, destination, resources, concurrency, and network. We do not recommend increasing concurrency or resources before identifying the bottleneck.

The window wait time can be used as the first-level indicator:

After you identify the bottleneck location, you can further troubleshoot based on the specific symptom:

• For message sources such as Kafka, DataHub, and LogHub, pay close attention to the number of partitions or shards. A single partition or shard can typically be consumed by only one concurrent thread. When data is concentrated in a few partitions, simply increasing total concurrency may not help. You can check the cumulative bytes of different Reader threads combined with source monitoring to determine whether partition or shard hotspots exist.

• For database log-based sources such as MySQL, pay close attention to large transactions, high-volume DML, frequent DDL, and Binlog growth rate. If synchronization speed is not high but latency continues to increase, check the source database audit logs, CPU, I/O, connections, and Binlog growth to determine whether the source is under pressure or whether unsynchronized databases and tables are generating a large volume of Binlog entries.

• When writing to MaxCompute, if the log shows uploader map size has reached uploaderMapMaximumSize, this typically indicates that the number of dynamic partition values within a single Flush interval is too large. Prioritize adjusting the partition granularity or dynamic partition columns to avoid using high-cardinality columns such as second-level timestamps, order IDs, or user IDs as partition values. Then evaluate whether you need to increase concurrency or resources.

DDL event processing errors

When a task fails, latency increases, or destination schema becomes inconsistent after a DDL event, first check whether the current channel supports the DDL type and what the task's DDL handling policy is.

Troubleshooting focus:

• Identify which DDL operations occurred at the source, such as adding columns, dropping columns, changing types, renaming, creating tables, dropping tables, or modifying indexes.

• Check whether the destination supports the same DDL or an equivalent schema change.

• Check whether the task policy is set to process normally, ignore, alert, or stop.

• Check whether the destination account has ALTER TABLE permissions.

• Check whether column mappings, primary keys, and partitions remain consistent before and after the DDL.

We do not recommend directly ignoring unsupported DDL operations. Ignoring a DDL may allow the task to continue running, but subsequent DML writes may continue to fail due to column mismatches, primary key changes, or destination schema inconsistencies.

Update or Delete not producing expected results

When destination data does not match expectations after Update or Delete operations, first check whether the destination has a primary key or unique key that can locate records, and whether the source and destination primary key mappings are consistent.

Common causes include:

• The destination does not have a primary key or unique key, making it impossible to update or delete by record.

• The source primary key and destination primary key are inconsistent.

• Primary key columns have been transformed, renamed, or written as null.

• The destination write mode does not support updates or deletes.

• The task is configured to ignore Delete operations or uses a special write policy.

• Write failures are recorded as dirty data, but the task continues running.

To troubleshoot, first use a small set of sample records to verify the source events, task mappings, and destination results, then decide whether to adjust primary keys, write modes, or reinitialize the destination table.

Increasing dirty data

When dirty data increases, do not directly increase the dirty data threshold. First determine whether the dirty data causes missing data, null columns, column truncation, primary key conflicts, or partition anomalies at the destination.

Troubleshooting focus:

• Check whether column types, lengths, precisions, and NOT NULL constraints match.

• Check whether destination primary keys, unique keys, or partition columns meet the requirements.

• Check whether the source has introduced excessively long strings, invalid encodings, special JSON, unparsable timestamps, or unsupported data types.

• Check whether the destination has throttling, write failures, or server-side filtering.

Only consider temporarily increasing the dirty data threshold when you have confirmed that the business can accept discarding these abnormal records. Follow up by fixing the source data or destination schema.