Check node

Overview

A Check node can monitor MaxCompute partitioned tables, FTP files, OSS files, HDFS files, OSS-HDFS files, and real-time synchronization tasks. A typical scenario is as follows:

A task in the DataWorks scheduling system needs to access an external database. Because the database write job runs outside of DataWorks, DataWorks cannot determine when the data is fully written and ready to be read. If the DataWorks task attempts to read the data before the write job finishes, it may read an incomplete dataset or the task may fail. To ensure that the DataWorks task runs as expected, you can use a Check node to monitor the table, file, or synchronization task. This ensures that downstream nodes can read the complete dataset.

Limitations

This feature is available only in DataWorks Professional Edition or later.

Prerequisites

• The RAM user that you want to use is added to your workspace.

  If you want to use a RAM user to develop tasks, you must add the RAM user to your workspace as a member and assign the Develop or Workspace Administrator role to the RAM user. The Workspace Administrator role has more permissions than necessary. Exercise caution when you assign the Workspace Administrator role. For more information about how to add a member and assign roles to the member, see Add members to a workspace.

• A serverless resource group is associated with your workspace. For more information, see the topics in the Use serverless resource groups directory.

• Before you develop a Check node, create the node first. For more information, see Create a node in a scheduled workflow.

• Before you use a Check node to validate an object, you must first create a corresponding data source for the target object.

  Object type	Preparations	References
  MaxCompute partitioned table	A MaxCompute data source is created and associated with DataStudio. To access data in a MaxCompute project, you must first register it as a MaxCompute data source in DataWorks. A MaxCompute partitioned table is created.	Associate MaxCompute compute resources Create a workspace and associate compute resources Create and use MaxCompute tables
  FTP file	An FTP data source is created. To access data from an FTP service, you must first register it as an FTP data source in DataWorks.	FTP data source
  OSS file	An OSS data source is created, and its access mode is set to AccessKey. To access data in an OSS bucket, you must first register it as an OSS data source in DataWorks. Note Check nodes currently support only OSS data sources that use the AccessKey authentication mode. Check nodes do not support OSS data sources that use RAM role authorization.	Create a bucket OSS data source
  HDFS file	An HDFS data source is created. To access data in an HDFS file system, you must first register it as an HDFS data source in DataWorks.	HDFS data source
  OSS-HDFS file	An OSS-HDFS data source is created. To access data from an OSS-HDFS service, you must first register it as an OSS-HDFS data source in DataWorks.	OSS-HDFS data source
  DLF table	A Data Lake Formation data source is created. Currently, only partitioned tables of the Paimon type are supported.	Data Lake Formation data source

• When you use a Check node to verify a real-time synchronization task, only tasks that synchronize data from Kafka to MaxCompute are supported. Before you use a Check node, create the corresponding real-time synchronization task. For more information, see Real-time synchronization task configuration (Old version).

Usage notes

• If an upstream delay causes a Check node to start after its configured check deadline, the Check node still runs but performs the check only once.

• The maximum runtime of a Check node is 24 hours.

Step 1: Develop the Check node

1. Open the Check node you created and configure it as prompted. A Check node can check a data source or a real-time synchronization task.

   Data source

   If you are checking an object via a data source, refer to the following table for the configuration parameters.

   Parameter	Description
   Check object	Data Source
   Data source type	The following data source types are supported: MaxCompute OSS FTP HDFS OSS-HDFS DLF
   Data source name	Select a data source that matches the selected data source type. If the required data source is not listed, click New Data Source on the right.
   Table name/path	If the data source type is MaxCompute or DLF, check by table name. Note Only partitioned tables are supported. If the data source type is OSS, FTP, HDFS, or OSS-HDFS, specify the path of the target file.
   Success condition	If the object is a table, you can check for the existence of a partition or for a period of no updates to the table's LastModifiedTime property. For files, the only available check is for file existence.
   Stop policy	Stop Time: The node performs checks at the configured interval until the specified time. Stop After N Checks: The node performs checks at the configured interval until it reaches the specified number of checks. Note The task runs based on the stop policy. If the success condition is not met when the stop condition is triggered, the Check node fails.

   Real-time synchronization task

   If the object to check is a real-time synchronization task, refer to the following table for the configuration parameters.

   Parameter	Description
   Check object	Real-time synchronization task
   Real-time synchronization task	Select an existing real-time synchronization task.
   Stop policy	Stop Time: The node performs checks at the configured interval until the specified time. Stop After N Checks: The node performs checks at the configured interval until it reaches the specified number of checks. Note The task runs based on the stop policy. If the success condition is not met when the stop condition is triggered, the Check node fails.

2. After you configure the check policy, configure scheduling properties for the node. For more information, see Configure scheduling properties for a node.

Step 2: Deploy and operate the node

1. After you configure the scheduling properties, submit and deploy the Check node to the production environment. For more information, see Deploy nodes and workflows.

2. After a task is deployed, it runs based on its configured schedule. You can view the deployed scheduled task in Operation and Maintenance Center > Node O&M > Auto Triggered Task O&M > Auto Triggered Task and perform maintenance operations. For more information, see Get started with Operation Center.