DocsICP FilingConsole
Documentation
Home Page DataWorks User Guide Data Studio (new version) Node development Node types General nodes OSS object inspection node

OSS object inspection node

Updated at:
Copy as MD
Product
My Favorites

An OSS object inspection node periodically checks for a specified object. If the object is found, the node triggers its downstream tasks. If the object is not found, the node continues to check at a configured interval until a stop condition is met. This node is typically used as a signaling mechanism between the DataWorks scheduling system and external scheduling systems. This topic describes how to use an OSS object inspection node and its key considerations.

Overview

A common use case for the OSS object inspection node is to manage dependencies on external data. For example, a DataWorks task may need to access an external database, but the task that writes data to that database is managed by an external scheduling system. In this scenario, DataWorks cannot determine when the write operation is complete. If the DataWorks task reads the data before the write operation finishes, it may read incomplete data or fail to read the data. To solve this, the external system can be configured to create a marker file, such as a .done file, in a specified file system after the write task is complete. Then, you can configure an OSS object inspection node in DataWorks to periodically check for this .done file. Once the node detects the file, it confirms that the data is ready and triggers the downstream tasks that need to access the database.

Note

  • You can configure the external scheduling system to generate the marker file in a file system of your choice.

  • This topic uses a .done file as an example. You can customize the name and format of your marker file to fit your business requirements.

image

  1. An external scheduling system detects that data in an external database is ready for access (for example, a write job is complete). It then creates a marker file, such as XXXX2024-09-29.done, in a specified file system. This topic uses a file with a .done suffix as an example, but you can define a custom file format to meet your business requirements.

  2. The OSS data source reads the marker file from the file system.

  3. The OSS object inspection node periodically checks the OSS data source for the marker file based on its configured detection policy.

    • If the node finds the marker file, this indicates the data is ready to be accessed. The OSS object inspection node then sends a success signal to its downstream nodes.

    • If the marker file is not found, it indicates that the data in the external database is not yet ready. The OSS object inspection node does not send a signal to its downstream nodes. It continues to check based on the configured policy until the retry limit is reached.

  4. The downstream node uses the result from the OSS object inspection node to decide whether to start its task.

    • If the OSS object inspection node reports a success, the downstream node starts its task to access the data from the external database.

    • If the OSS object inspection node fails to detect the file and times out, the downstream node does not start its task.

  5. The downstream node accesses the data in the external database.

Note

External databases can include but are not limited to Oracle, MySQL, SQL Server, and other database or storage services.

Prerequisites

  • The RAM user that you want to use is added to your workspace.

    If you want to use a RAM user to develop tasks, you must add the RAM user to your workspace as a member and assign the Develop or Workspace Administrator role to the RAM user. The Workspace Administrator role has more permissions than necessary. Exercise caution when you assign the Workspace Administrator role. For more information about how to add a member and assign roles to the member, see Add members to a workspace.

  • A serverless resource group is associated with your workspace. For more information, see the topics in the Use serverless resource groups directory.

  • Create an OSS object inspection node. For more information, see Create a scheduling workflow node.

Usage notes

  • When the task runs, the node uses STS mode authorization to check the OSS object. Make sure the required OSS bucket permissions are configured.

  • In the development and production environments, the node uses an access identity with the permissions described in Responsibilities of a workspace administrator. Ensure that the OSS bucket permissions are configured correctly.

Step 1: Configure the node

After you create an OSS object inspection node, open it and follow the on-screen instructions to configure its properties.

  1. Configure the object and detection policy.

    • OSS Object: Enter the path of the OSS object to check. Scheduling parameters such as ${bizdate} and ${cyctime} are supported.

    • Timeout: The maximum time to wait for the object. If the object is not found within this period, the node stops checking and the task fails.

  2. After you configure the node, configure its scheduling properties. For more information, see Configure scheduling properties for a node.

Step 2: Deploy and operate the node

  1. After you configure the scheduling properties, deploy the node to the production environment. For more information, see Deploy nodes and workflows.

  2. After you deploy a node, it runs periodically based on its scheduling properties. You can view and manage deployed tasks on the Operation and Maintenance Center > Node O&M > Auto Triggered Task O&M > Auto Triggered Task tab. For more information, see Introduction to Operation Center.

Previous：Parameter nodeNext：Python node