System time jump drill

Updated at:

Verify that your system can detect and recover from sudden clock changes before they disrupt time-dependent services such as logging, synchronization, and backup.

Why run time jump drills?

A sudden time jump can cause issues such as:

  • Certificate validation failures -- TLS/SSL certificates may appear expired or not yet valid.

  • Cron job scheduling errors -- Scheduled tasks may execute at the wrong time or be skipped entirely.

  • Log ordering problems -- Out-of-sequence timestamps can make log analysis and incident investigation unreliable.

  • Synchronization and backup failures -- Replication and backup processes that depend on timestamps may fail or produce inconsistent results.

A time jump drill confirms that your system can detect, correct, and recover from an unexpected clock change before one occurs in production.

How it works

The Cloud Assistant plugin ACS-ECS-TimeJump adjusts the time forward or backward by setting a system time offset. Fault recovery relies on the chrony service synchronizing with NTP servers, so chrony must be installed on the instance before fault injection.

Prerequisites

Procedure

Fault injection

  1. Connect to the ECS instance. See Use Workbench to connect to a Linux instance through SSH.

  2. With sudo privileges, run the Cloud Assistant plugin ACS-ECS-TimeJump:

    sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params inject,[time=paramA],[duration=paramB]

    Parameters in [] are optional. Parameter descriptions:

    Parameter Required Description Default
    time No Time offset in seconds. Positive values adjust the time forward; negative values adjust the time backward. 1
    duration No Duration of fault injection, in seconds. 300

    The following output indicates that the plugin ACS-ECS-TimeJump has started.

    [ecs-user@iZ0jlfp0by6c          ~]$ sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params inject,time=120
    Will jump 120 seconds.
    Wed Sep 18 02:34:18 PM CST 2024
    Done
  3. Run date to check whether the current system time meets your expectations.

    [ecs-user@iZ0jlfp0by6d    ~]$ date
    Wed Sep 18 02:34:55 PM CST 2024

Fault recovery

Recover from the time jump in either of the following ways:

  • Method 1: Wait for the duration to expire for automatic recovery.

  • Method 2: Run the recovery command on the ECS instance:

    sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params recover
Important

The recovery command triggers a time synchronization. Chrony synchronization time depends on multiple factors. Verify that the time has been recovered after the drill ends.

Drill example

This example injects a time jump of 1800 seconds (30 minutes) and then recovers the system time.

  1. Run the following script to print the current system time every second, so you can observe the time jump in real time:

    while true; do
           date +"%Y-%m-%d %H:%M:%S"
           sleep 1
       done
  2. Open a second terminal and inject a fault to simulate a time jump. This command adjusts the system time forward by 1800 seconds. The output includes the adjusted system time.

    sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params inject,time=1800
    [ecs-user@iZ0jlfp0by6xxx ~]$ sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params inject,time=1800
    Will jump 1800 seconds.
    Wed Sep 18 04:05:02 PM CST 2024
    Done
  3. Verify the fault injection. In the first terminal, the system time jumps at 15:35:01.

    [ecs-user@iZ0jlfp0by6xxx          ~]$ while true; do
    >    date +"%Y-%m-%d %H:%M:%S"
    >    sleep 1
    > done
    2024-09-18  15:34:56
    2024-09-18  15:34:57
    2024-09-18  15:34:58
    2024-09-18  15:34:59
    2024-09-18  15:35:00
    2024-09-18  15:35:01
    2024-09-18  16:05:02
    2024-09-18  16:05:03
    2024-09-18  16:05:04
    2024-09-18  16:05:05
    2024-09-18  16:05:06
    2024-09-18  16:05:07
    2024-09-18  16:05:08
    2024-09-18  16:05:09
  4. Run the recovery command. The system synchronizes time with the NTP server and returns to normal.

    sudo acs-plugin-manager --exec --plugin ACS-ECS-TimeJump --params recover
    2024-09-18 16:08:45
    2024-09-18 16:08:46
    2024-09-18 16:08:47
    2024-09-18 16:08:48
    2024-09-18 16:08:49
    2024-09-18 16:08:50
    2024-09-18 15:39:01
    2024-09-18 15:39:02
    2024-09-18 15:39:03
    2024-09-18 15:39:04
    2024-09-18 15:39:05
    2024-09-18 15:39:06
    2024-09-18 15:39:07
    2024-09-18 15:39:08