This topic describes the core concepts of task scheduling.
Procedure
A simple job can be split into multiple steps. These steps are executed sequentially by different handlers. Each step starts only after the previous one completes successfully.
For example, a reconciliation job involves the following steps: downloading the bank's settlement file, parsing the file based on a predefined format, comparing the file data with the institution's daily transaction data, and processing the results, such as generating and handling discrepancies. A dedicated handler can execute each step independently.
Task sharding
Task sharding is useful for jobs with a long running time. When the data volume is large, the scheduling system can split the data into multiple data chunks within a single step. This allows the job data to be distributed across different machines for execution.
While each data chunk is executed by a single thread, multiple chunks can be scheduled to run concurrently on multiple machines or on multiple threads on a single machine. The scheduling center adjusts the scheduling method based on the current system load.
Task sharding shortens a job's running time, helps overcome the computing limits of a single machine, and reduces the impact of partial failures on the system.
Task topology
A task topology is a collection of event-triggered tasks that defines their dependencies. Subtasks can run concurrently or based on specific conditions. In a task topology, the execution flow for all tasks starts at a start node and ends at a stop node. This flow forms a directed acyclic graph.