System time jump drill
Verify that your system can detect and recover from sudden clock changes before they disrupt time-dependent services such as logging, synchronization, and backup.
Why run time jump drills?
A sudden time jump can cause issues such as:
-
Certificate validation failures -- TLS/SSL certificates may appear expired or not yet valid.
-
Cron job scheduling errors -- Scheduled tasks may execute at the wrong time or be skipped entirely.
-
Log ordering problems -- Out-of-sequence timestamps can make log analysis and incident investigation unreliable.
-
Synchronization and backup failures -- Replication and backup processes that depend on timestamps may fail or produce inconsistent results.
A time jump drill confirms that your system can detect, correct, and recover from an unexpected clock change before one occurs in production.
How it works
The Cloud Assistant plugin ACS-ECS-TimeJump adjusts the time forward or backward by setting a system time offset. Fault recovery relies on the chrony service synchronizing with NTP servers, so chrony must be installed on the instance before fault injection.
Prerequisites
-
Cloud Assistant Agent is installed on the ECS instance.
-
The status of Cloud Assistant is Normal on the ECS instance. See View the status of Cloud Assistant and handle anomalies.
-
The NTP time synchronization service is configured and running. See Manage time synchronization service.
Procedure
Fault injection
-
Connect to the ECS instance. See Use Workbench to connect to a Linux instance through SSH.
-
With sudo privileges, run the Cloud Assistant plugin
ACS-ECS-TimeJump:sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params inject,[time=paramA],[duration=paramB]Parameters in
[]are optional. Parameter descriptions:Parameter Required Description Default timeNo Time offset in seconds. Positive values adjust the time forward; negative values adjust the time backward. 1 durationNo Duration of fault injection, in seconds. 300 The following output indicates that the plugin
ACS-ECS-TimeJumphas started.[ecs-user@iZ0jlfp0by6c ~]$ sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params inject,time=120 Will jump 120 seconds. Wed Sep 18 02:34:18 PM CST 2024 Done -
Run
dateto check whether the current system time meets your expectations.[ecs-user@iZ0jlfp0by6d ~]$ date Wed Sep 18 02:34:55 PM CST 2024
Fault recovery
Recover from the time jump in either of the following ways:
-
Method 1: Wait for the duration to expire for automatic recovery.
-
Method 2: Run the recovery command on the ECS instance:
sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params recover
The recovery command triggers a time synchronization. Chrony synchronization time depends on multiple factors. Verify that the time has been recovered after the drill ends.
Drill example
This example injects a time jump of 1800 seconds (30 minutes) and then recovers the system time.
-
Run the following script to print the current system time every second, so you can observe the time jump in real time:
while true; do date +"%Y-%m-%d %H:%M:%S" sleep 1 done -
Open a second terminal and inject a fault to simulate a time jump. This command adjusts the system time forward by 1800 seconds. The output includes the adjusted system time.
sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params inject,time=1800[ecs-user@iZ0jlfp0by6xxx ~]$ sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params inject,time=1800 Will jump 1800 seconds. Wed Sep 18 04:05:02 PM CST 2024 Done -
Verify the fault injection. In the first terminal, the system time jumps at 15:35:01.
[ecs-user@iZ0jlfp0by6xxx ~]$ while true; do > date +"%Y-%m-%d %H:%M:%S" > sleep 1 > done 2024-09-18 15:34:56 2024-09-18 15:34:57 2024-09-18 15:34:58 2024-09-18 15:34:59 2024-09-18 15:35:00 2024-09-18 15:35:01 2024-09-18 16:05:02 2024-09-18 16:05:03 2024-09-18 16:05:04 2024-09-18 16:05:05 2024-09-18 16:05:06 2024-09-18 16:05:07 2024-09-18 16:05:08 2024-09-18 16:05:09 -
Run the recovery command. The system synchronizes time with the NTP server and returns to normal.
sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params recover2024-09-18 16:08:45 2024-09-18 16:08:46 2024-09-18 16:08:47 2024-09-18 16:08:48 2024-09-18 16:08:49 2024-09-18 16:08:50 2024-09-18 15:39:01 2024-09-18 15:39:02 2024-09-18 15:39:03 2024-09-18 15:39:04 2024-09-18 15:39:05 2024-09-18 15:39:06 2024-09-18 15:39:07 2024-09-18 15:39:08