To have a node generate and run instances immediately after it is deployed to the production environment, set its instance generation mode to Immediately After Deployment.
Background
After you deploy a node, you can view its latest configuration in Operation Center. Every night, DataWorks generates scheduled instances for the next day's schedule based on the configuration of each scheduled task. When you deploy a new or updated node to the production environment, the selected instance generation mode determines when the changes affect scheduled instances and their dependencies.
In DataWorks, the instance generation mode provides two options that control when your changes take effect: Next Day and Immediately After Deployment.
-
Next Day: Node creation and update operations affect the scheduled instances of the next day. If a task must run immediately after being deployed to the production environment, you can run a backfill data operation for the task.
-
Immediately After Deployment: Node creation and update operations take effect immediately. However, a time lag exists between deployment and when runnable instances are generated. This lag has different effects depending on the scenario. For more information, see Common scenarios for immediate instance generation.
Notes
-
If you set the instance generation mode to Immediately After Deployment, changes to rerun properties do not apply to instances that have already expired.
-
Nodes within a workflow cannot be individually configured for immediate generation. This option must be configured for the entire workflow on its scheduling configuration page.
-
Regardless of whether you choose Next Day or Immediately After Deployment in the scheduling configuration, the system generates all scheduled instances for the next day between 22:00 and 24:00 daily. Tasks deployed during this period will not generate instances until the day after tomorrow.
-
Inconsistent instance generation modes for upstream and downstream tasks may create isolated nodes.
-
Time lag for immediate instance generation: To prevent unexpected behavior, a 10-minute lag is built into the immediate generation process. The scheduled time of a task must be at least 10 minutes after the deployment time for the task to run with the latest configuration.
-
Scope of immediate instance generation (differences between task code and schedule settings): Not all changes take effect immediately. You need to distinguish between task code changes and schedule settings changes, as their effective logic differs.
-
Task code (such as SQL and Shell script logic): Instances read the latest code of the task at runtime. Therefore, if a code error causes a runtime exception, you typically only need to fix the code, redeploy it, and rerun the corresponding instances to recover. You do not need to regenerate instances.
-
Schedule settings (such as scheduling parameters and rerun policies): These properties are fixed in the instance at generation time. If you modify the schedule settings of a task after instances are generated, the existing instances are not automatically updated and will still run with the old configuration from when they were generated.
Example: If you change a scheduling parameter from
${bizdate}to${datehour}and deploy the change, the reference to${datehour}in the code takes effect immediately. However, the old parameter context fixed in the already-generated instances is not refreshed. This mismatch may cause parameter parsing failures at runtime. The same applies to scenarios such as modifying the data source associated with a node. Instances already generated for the current day will still run with the pre-change configuration.
To apply schedule settings changes to already-generated instances, you can:
-
In Operation Center, find the updated task and use the Refresh Instances operation to batch update the schedule settings of already-generated instances.
-
Or run a backfill data operation for the task. The backfill operation generates and runs instances based on the latest task configuration.
-
How immediate instance generation works
Immediate instance generation after deployment applies only to scheduled times in the future. Specifically, instances are executed only when the scheduled time of the task is later than the deployment time.
-
On the day a new task is created, scheduled instances are generated, but only instances whose scheduled time is in the future will be executed.
-
If you update the scheduled time of a node and the new scheduled time is in the past, no instance is generated. If the new scheduled time is in the future, new instances are generated based on the new configuration and replace the previous instances.
NoteThe scheduled time must be at least 10 minutes after the node deployment time for instances to be properly generated immediately.
Scheduled time falls within the normal execution range
-
Scenario 1: On the day a new node is created, real scheduled instances are generated. When the scheduled time is in the future relative to the instance generation time, and the difference between the two time points is greater than 10 minutes, the instance is scheduled and executed normally. For details, see Immediate instance generation after deploying a new node.
-
Scenario 2: After you update the node configuration, if the scheduled time is in the future relative to the instance generation time, and the difference between the two time points is greater than 10 minutes, the instance is scheduled and executed normally. The executed instance reflects the updated configuration. For details, see Update the schedule of a deployed task.
-
Scenario 4: Impact of changing the scheduled time on downstream dependencies.
-
If the new scheduled time of the instance is in the future relative to the instance generation time, and the difference between the two time points is greater than 10 minutes, the instance is scheduled and executed normally. Downstream instances that have not yet run will depend on the new instances after the change. For details, see Impact of schedule time changes on downstream dependencies.
-
If the new scheduled time of the instance is in the past relative to the instance generation time, dry-run instances are generated, and downstream instances that have not yet run become isolated nodes. For details, see Inconsistent instance generation modes for upstream and downstream tasks.
-
We recommend that you do not use this feature when modifying the schedule settings of production nodes. This feature may cause dependency changes, dependency inconsistencies, instance replacement, or instance deletion, which can make the dependencies complex for the current day. However, the task dependencies will return to normal the next day.
Scheduled time falls within the dry-run range
If the scheduled time is in the past relative to the node deployment time, scheduled instances are still generated, but the instances dry run. The instance status is Expired instance generated in real time, and no actual code logic is executed. For details, see Immediate instance generation after deploying a new node.
-
Scenario 1: The scheduled time is in the future relative to the instance generation time, but the difference between the two time points is less than 10 minutes. The instance status is Expired Instance Generated in Real Time.
Example: Node A has a scheduled time of
09:05, and the node is deployed at09:00. If the instance's scheduled time is in the future relative to the instance generation time, but the difference between the two time points is less than 10 minutes, node A generates a dry-run instance with the status Expired Instance Generated in Real Time. -
Scenario 2: The scheduled time is in the past relative to the instance generation time. An Expired Instance Generated in Real Time instance is immediately generated.
Example: Node A has a scheduled time of
09:00and is deployed at10:00. The instance generation time is before the deployment time. Node A immediately generates a dry-run instance with the status Expired Instance Generated in Real Time.
Common scenarios for immediate instance generation
When you use the Instant generation after publishing mode to generate instances, the instance execution and upstream/downstream dependency behavior in related scenarios is as follows:
Update the schedule of a deployed task
After you update the scheduled time of a production task and deploy it, instances from both before and after the change may coexist on the same day, which can make dependencies complex. Unless necessary, we recommend that you do not use the Instant generation after publishing mode for deployed tasks. The following is an example scenario where the schedule is changed from hourly to daily.
This scenario occurs only on the day the task with immediate instance generation is deployed. The next day, instances are generated normally based on the configuration.
-
Case 1: Change from every-6-hour scheduling to daily scheduling, where the daily scheduled time is in the past.
At 09:00, modify the task scheduled time to a past time by changing the schedule from every 6 hours to daily at
08:00. The instance dependencies for the current day are as follows:
-
Case 2: Change from every-6-hour scheduling to daily scheduling, where the daily scheduled time is in the future.
At 09:00, modify the task scheduled time to a future time by changing the schedule from every 6 hours to daily at 18:00. The instance dependencies for the current day are as follows:
-
Generate instance: A daily instance A3 is generated after 09:00 based on the new configuration.
-
Replace existing instances: The new instance A3 replaces the original instances A3 and A4.
-
Retain instances: Hourly instances before 09:10 are retained.
-
-
The scheduled time is in the future: DataWorks replaces the already-generated instances in the future time range based on the latest schedule settings.
-
The scheduled time is in the past: DataWorks retains instances before the effective time of the modified configuration and replaces or deletes instances after the effective time.
After the scheduled time is changed and deployed, whether instances are regenerated and actually executed on the current day depends on the task's scheduled time setting and the time when the change is deployed to the production environment. For details, see Background.
Impact of schedule time changes on downstream dependencies
For instances immediately generated after task deployment, downstream tasks set dependencies based on the latest schedule settings of the upstream task. The schedule can be daily, monthly, or hourly.
For production tasks with changed scheduled times, downstream instances set dependencies for both newly generated instances and unreplaced old instances based on the latest schedule settings. For details about dependency behavior in hourly and minutely scenarios, see Dependency details for schedule time changes. This scenario occurs only when the node version to be deployed has its instance generation mode set to Instant generation after publishing and the scheduled time is changed.
The following are example scenarios:
-
Case 1: The upstream node's schedule is changed from every 6 hours to every 8 hours, and immediate instance generation is selected.
-
Case 2: The upstream node's schedule is changed from every 6 hours to daily at 16:00, and immediate instance generation is selected.
After you change the task scheduled time and select immediate instance generation, downstream instances adjust their dependencies based on the latest schedule settings to ensure that the dependency logic between newly generated instances and unreplaced old instances is correct. For both unreplaced old instances and newly generated instances, downstream tasks set the corresponding dependencies to ensure correct scheduling logic.
Inconsistent instance generation modes for upstream and downstream tasks
If both upstream and downstream nodes are newly created and their instance generation modes are inconsistent — for example, the upstream uses T +1 generated next day and the downstream uses Instant generation after publishing — this results in isolated nodes. Isolated nodes are not automatically scheduled. If such an isolated node has many downstream dependencies, this may cause significant issues.
