Data Integration settings

How it works

The Processing Policy for DDL Messages in Real-time Sync feature lets you preset a default DDL (Data Definition Language) synchronization policy for different destination data source types, such as MaxCompute and Hologres. Once saved, the system automatically applies this policy when you create a new real-time synchronization task for an entire database.

1. Select a destination type: From the Synchronize To drop-down menu, select a destination data source type.

2. Set the synchronization policy: For different types of source DDL change messages, such as adding a column or clearing a table, configure and save the corresponding processing method.

   • Normal treatment: The DDL change is applied to the destination schema.

     Important

     When a new column is added at the source and synchronized to the destination, the system does not backfill historical data into the newly created column.

   • Ignore: The system discards the DDL event, and the destination schema remains unchanged.

   • Error: The real-time synchronization task stops, and the system sets its status to Error.

3. After you save the policy, the system automatically applies it when you create another real-time synchronization task for an entire database with the same destination type.

4. To override the policy for a single task, click Advanced Configuration > DDL Capability Configuration in the upper-right corner of the task details page to configure a task-specific DDL message processing policy.

Recommendations

For real-time synchronization from MySQL and PolarDB for MySQL sources, add new columns at the end of the table, not in the middle. If you must add a column in the middle of a source table, follow these constraints:

• During the full initialization phase, do not add columns in the middle of a table. Doing so causes data anomalies in the subsequent real-time synchronization phase.

• During the real-time synchronization phase, you must reset the synchronization offset to a timestamp after the column was added. Otherwise, data anomalies will occur in subsequent synchronization.

If a data anomaly occurs, you can edit the task to remove the table with the added column, and then save the task to restore the data.

Use cases

• Standardize development: Save common task configurations as templates for team members to reuse, enforcing consistent development practices.

• Improve development efficiency: When creating new tasks, you can start from an existing template to accelerate development and reduce repetitive work.

• Preserve knowledge: Save complex or high-quality task implementations as templates to facilitate knowledge sharing and transfer.

Main operations

• View the template list: This list shows all available task templates in the current project, including their template ID, name, and last update time.

• Search for templates: Use the search box at the top to quickly find a template by its name, ID, or related data source information.

• Manage templates: In the Operation column of the list, you can delete templates.

Use cases

• Standardize monitoring: Define uniform alert standards for different task types (such as critical and routine tasks) or alert scenarios (such as errors, slowdowns, or non-execution).

• Improve configuration efficiency: When you configure alerts for a task, you can directly reference a public rule instead of repeatedly entering information such as notification methods and contacts.

• Centralize management: Manage all alert rules in a unified view, making it easy to modify notification recipients or adjust alert thresholds in batches.

Main operations

• Create a new rule: Click the Create button to create a Public Alert Rule. Define the Alert Rule Name, select an Alert Reason (such as business latency, dirty data, or failover), configure the Alert Notification Method (such as phone call, SMS, email, DingTalk, webhook, or Lark), and set the Configure Advanced Parameters. For more information about alert configuration, see Best practices for configuring alerts for real-time synchronization tasks.

  Note

  • For anomalies related to business latency, task status, and task resource utilization, you can also enable recovery notification to inform recipients when the status returns to normal.

  • Webhook and Lark are available only in the Enterprise Edition.

• Task Reference: Click Task Reference to efficiently apply the alert rule to, or remove it from, multiple tasks. You can also enable or pause alerts for these tasks in batches to avoid repetitive operations.

• Test: Send a test alert to verify that the alert configuration is correct and that notifications are delivered to the specified recipients.

  Note

  This feature sends a live alert notification. Use it with care to avoid causing unnecessary interruptions to others.