The Time window MR component extends the standard MaxCompute MapReduce (MR) component with multi-date loop execution: it runs day-level MR tasks across a range of historical dates in parallel, eliminating the need to create a separate task for each day. A common use case is running EasyRecFGMapper tasks for the past 30 days in parallel when building a recommendation algorithm pipeline.
Limitations
-
Multi-date loop execution supports day-level data backfill only.
-
Before scheduling the pipeline with Periodic Scheduling, disable multi-date loop execution. This prevents unintended backfill tasks from running in production.
-
The Maximum concurrency setting applies only to the node it is configured on. If multiple nodes run backfill simultaneously, factor in the total concurrency capacity of your project resources.
How it works
Each subtask in the loop is assigned a cycle date — the business date that subtask processes, expressed in the format you choose. The system variable ${pai.system.cycledate} is replaced with that date at runtime, so you reference it in your MR task input parameters to target the correct day's data.
For example, with Business base date 20230210 and Date format yyyy-MM-dd, the variable resolves to 2023-02-10 for the subtask that processes February 10, 2023.
Configure the component
Configure the component parameters only in Designer on the Parameters Setting tab. Multi-date loop execution is enabled by default.
| Tab | Parameter | Description |
|---|---|---|
| Parameters Setting | Business base date | The reference date for calculating the time window. Enter a date directly (for example, 20230210), or reference a global variable. See Advanced feature: global variable. |
| Enable multi-date loop execution | Enabled by default. When disabled, the component behaves as a standard MR component. | |
| Execution time window | Defines which dates to process, expressed as offsets from the business base date. Supports integers, open intervals (, and closed intervals ]. Separate multiple intervals with a comma. The system generates one subtask per date in the window, up to 100 subtasks. |
|
| Maximum concurrency | The maximum number of subtasks that run in parallel. Default: 3. This setting applies only to the current node, not to other nodes in the pipeline — increase with care if other nodes are running backfill simultaneously. |
|
| Date format | The format used to generate ${pai.system.cycledate}. Options: yyyyMMdd (default), yyyy-MM-dd, yyyy/MM/dd. |
|
| Resource OSS path | The OSS directory where the resource file is located. | |
| Resource file list | The resource files to include. Separate multiple files with commas (,). |
|
| OSS path for classpath | The OSS path of the JAR file. | |
| Main class | The fully qualified name of the main class. | |
| MR task input parameters | Reference ${pai.system.cycledate} in the date field so each subtask processes the correct business date. |
Time window syntax
The Execution time window uses relative day offsets from the Business base date. Parentheses ( mark open (excluded) boundaries; square brackets ] mark closed (included) boundaries. Combine intervals and single-point values with commas.
For example, if you set the Execution time window to (-4,-2],0 and the business base date is 20230210, tasks for February 7, 2023, February 8, 2023, and February 10, 2023 are executed.
Up to 100 subtasks can run per execution.
Example
For a complete end-to-end example using the 2_rec_sln_demo_dssm_recall_vector_recall_sample_fg_encoded_v1 pipeline, see Vector Recall.
What's next
-
Advanced feature: global variable — use global variables to parameterize the business base date across pipelines.
-
Vector Recall — see a full recommendation pipeline that uses this component for multi-day feature generation.