Check node

Overview

A Check node can monitor MaxCompute partitioned tables, FTP files, OSS files, HDFS files, OSS_HDFS files, and real-time synchronization tasks. A typical scenario is as follows:

A task in the DataWorks scheduling system needs to access an external database. Because the write job for that database runs outside of DataWorks, DataWorks cannot determine when the data is fully written and ready to be read. If a DataWorks task reads the data before the write job finishes, it may read an incomplete dataset or fail. To ensure that downstream tasks run correctly, you can use a Check node to monitor the table, file, or synchronization task and confirm that the data is ready for consumption.

Limits

This feature is available only in DataWorks Professional Edition or later.

Prerequisites

• The RAM user that you want to use is added to your workspace.

  If you want to use a RAM user to develop tasks, you must add the RAM user to your workspace as a member and assign the Develop or Workspace Administrator role to the RAM user. The Workspace Administrator role has more permissions than necessary. Exercise caution when you assign the Workspace Administrator role. For more information about how to add a member and assign roles to the member, see Add members to a workspace.

• A serverless resource group is associated with your workspace. For more information, see the topics in the Use serverless resource groups directory.

• Before you develop a Check node, create the node. For more information, see Create nodes for scheduled workflows.

• When you use a Check node to validate a data source, create the corresponding data source before you use the Check node. The following table describes the required preparations.

  Object type	Preparations	References
  MaxCompute partitioned table	A MaxCompute data source is created and associated with DataStudio. To access data in a MaxCompute project, you must first register the project as a MaxCompute data source in DataWorks. A MaxCompute partitioned table is created.	Associate a MaxCompute compute resource Create and use MaxCompute tables
  FTP file	An FTP data source is created. To access data from an FTP service, you must first register the service as an FTP data source in DataWorks.	FTP data source
  OSS file	An OSS data source is created with the access mode set to AccessKey. To access data in an OSS bucket, you must first register the bucket as an OSS data source in DataWorks. Note Check nodes currently support only OSS data sources that use the AccessKey authentication mode. OSS data sources configured with RAM role authorization cannot be used in Check nodes.	Create a bucket OSS
  HDFS file	An HDFS data source is created. To access data in an HDFS file system, you must first register the file system as an HDFS data source in DataWorks.	HDFS
  OSS_HDFS file	An OSS_HDFS data source is created. To access data from an OSS_HDFS service, you must first register the service as an OSS_HDFS data source in DataWorks.	OSS-HDFS
  DLF table	A Data Lake Formation data source is created. Only Paimon partitioned tables are supported.	Data Lake Formation data source

• When you use a Check node to verify a real-time synchronization task, only Kafka-to-MaxCompute real-time synchronization tasks are supported. Create the corresponding real-time synchronization task before you use the Check node. For more information, see Configure a real-time synchronization task (legacy).

Usage notes

• If an upstream delay causes a Check node to start after its configured check deadline, the Check node still runs but performs the check only once.

• The maximum runtime of a Check node is 24 hours.

Step 1: Develop the Check node

1. Open the Check node you created and configure it as prompted. A Check node can check a data source or a real-time synchronization task.

   Data source

   When the check object is a data source, configure the parameters described in the following table:

   Parameter	Description
   Check object	Data Source
   Data source type	The following data source types are supported: MaxCompute OSS FTP HDFS OSS_HDFS DLF
   Data source name	Select a data source that matches the selected data source type. If the required data source is not listed, click New Data Source on the right.
   Table name/Path	If the data source type is MaxCompute or DLF, specify a table name for the check. Note Only partitioned tables are supported. If the data source type is OSS, FTP, HDFS, or OSS_HDFS, specify the path of the target file.
   Success condition	If the check object is a table, you can check for the existence of a partition or for a sustained period of no updates to the LastModifiedTime property. If the check object is a file, you can only check whether the file exists.
   Stop policy	Stop time: The node performs checks at the configured interval until the specified time. Stop after N checks: The node performs checks at the configured interval until it reaches the specified number of checks. Note The task runs based on the stop policy. If the success condition is not met when the stop condition is reached, the Check node automatically exits and enters a failed state.

   Real-time synchronization task

   When the check object is a real-time synchronization task, configure the parameters described in the following table:

   Parameter	Description
   Check object	Real-time synchronization task
   Real-time synchronization task	Select an existing real-time synchronization task.
   Stop policy	Stop time: The node performs checks at the configured interval until the specified time. Stop after N checks: The node performs checks at the configured interval until it reaches the specified number of checks. Note The task runs based on the stop policy. If the success condition is not met when the stop condition is reached, the Check node automatically exits and enters a failed state.

2. After you configure the check policy, configure scheduling properties for the node. For more information, see Configure node scheduling.

Step 2: Deploy and operate the node

1. After you configure the scheduling properties, submit and deploy the Check node to the production environment. For more information, see Node and workflow deployment.

2. After a task is deployed, it runs based on the configured schedule. You can view the deployed scheduled task in Operation and Maintenance Center > Node O&M > Auto Triggered Task O&M > Auto Triggered Task and perform O&M operations. For more information, see Get started with operation center.