Best practices for Check nodes in Data Integration real-time tasks-DataWorks(DataWorks)-阿里云帮助中心

Limitations

This feature supports real-time synchronization tasks created in Data Integration and DataStudio.
The following real-time synchronization channels are supported for Data Integration tasks:
- MySQL to MaxCompute
- Kafka to MaxCompute
- MySQL to Hologres

Serverless resource groups are supported.

If you encounter the following error while using a legacy resource group, we recommend upgrading to a serverless resource group. You can also submit a ticket to have technical support upgrade the underlying version of your legacy resource group.

java.lang.RuntimeException: unknown type : 241 at 
com.alibaba.dataworks.messagewrapper.typehandler.TypeHandlerFactory.hander(TypeHandlerFactory.java:50) at 
com.alibaba.dataworks.messagewrapper.Main.main(Main.java:57) at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at 
java.lang.reflect.Method.invoke(Method.java:498) at 
com.taobao.pandora.boot.loader.MainMethodRunner.run(MainMethodRunner.java:54) at 
com.taobao.pandora.boot.loader.Launcher.launch(Launcher.java:85) at 
com.taobao.pandora.boot.loader.Launcher.launch(Launcher.java:48) at 
com.taobao.pandora.boot.loader.SarLauncher.main(SarLauncher.java:167)

Node configuration

Usage notes

Always test tasks before running them in a production environment. Because underlying resource group versions can differ, you must verify that your synchronization task runs as expected.
The Check node uses only the task's scheduling time for verification. When an instance is rerun, it still uses the initial scheduling time of that instance, not the actual time of the rerun.
A successful check from the Check node indicates that the event time of data written from the source to the destination table has surpassed the scheduling time. You can then configure SQL tasks that depend on the destination table.

Create a Data Integration real-time synchronization task

Data Integration console

Log on to the DataWorks console. In the target region, click Data Integration > Data Integration in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Integration.
In the left-side navigation pane, click Synchronization Task. In the Synchronization Task section, select a data Source and Destination, and then click Create Sync Task.

Note
For information about supported synchronization channels, see Limitations.
Configure the synchronization type. Select a type that includes Real-time Synchronization, such as real-time synchronization of an entire MySQL database to MaxCompute.
Complete the remaining configurations for the synchronization task. For more information, see Common business scenarios and best practices.

DataStudio

Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
Click Create Node, select Real-time Synchronization, and specify a custom name for the node.
Drag the Input and Output components to the canvas and configure the real-time synchronization parameters. For more information, see Configure a real-time synchronization task (Old version).

Create a Check node

Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
Click Create Node, select Check Node, and specify a custom name for the node.
Configure the parameters for the Check node.
- Check Object: Select Real-time Synchronization Task.
- Real-time Synchronization Task: Select the task that you created in the Create a Data Integration real-time synchronization task step.
- Check Stop Policy: If the conditions specified in the check stop policy are not met, the Check node task automatically fails and exits. For more information about the configuration, see Step 2: Configure check policies.

Check result examples

Successful check

Task configuration: The task is scheduled to run every 5 minutes. The test instance is scheduled for 20241008143000 (UTC+8 timestamp: 1728369000000).

Set Check Object to Real-time Synchronization Task and select the task sync_mysql_to_odps_20241008_133744. For Check Stop Policy, select Number of Checks, and configure it to perform 2 checks at 5-minute intervals. The maximum runtime for the Check node task is 24 hours.
When the task latency is low, the Check node passes the check. Timestamp: 1728369386000.

The task logs show that after several polling attempts with cycTime=20241008143000 and fileId=504592170, the check completes. The shell command exits with code 0, the task status is FINISH, and the total duration is 630 seconds.

Failed check

High task latency can cause the check to fail after two attempts, leading to the failure of the entire task.

The logs show multiple java.net.SocketTimeoutException: Read timed out errors, eventually resulting in a Shell run failed message. The exit code is 100, and the duration is approximately 1030 seconds.