Persist startup configurations for ECS instances

Updated at:

Configure persistent disk mounts, process auto-start, and monitoring for ECS instances to survive unexpected restarts.

Use cases

ECS instances may restart unexpectedly due to OOM errors or host failures. Temporary configurations—such as manually mounted disks or manually started processes—are lost after a restart, causing data disk unavailability and extended downtime.

Solution architecture

This solution converts temporary operations into permanent settings so the instance automatically restores its business state after a restart. It combines OS-native features with Alibaba Cloud monitoring to enable automatic recovery and alerting.

  • Persistent disk mounting: Use /etc/fstab to record disk mount information with the device UUID instead of the device name (such as /dev/vdb). This avoids mount failures caused by device name changes after a restart.

  • Persistent business process management: Use systemd to manage business processes as system services. systemd starts services at boot and restarts them on unexpected exit.

  • Status monitoring and validation:

    • Monitoring: Configure Cloud Monitor to track disk mount points, core processes, and instance restart events. Receive alerts when automatic recovery fails.

    • Validation: Use the Cloud Assistant fault injection plug-in to simulate crashes in non-production environments and verify that configuration persistence and alerting work as expected.

Procedure

Complete disk mounting, process auto-start, monitoring, and validation in the following steps.

Configure automatic disk mounting at boot

Use the disk UUID in /dev/vdb1's fstab entry to ensure automatic mounting after restarts. UUIDs remain constant across the instance lifetime, unlike device names.

  1. Back up the fstab file

    Back up /etc/fstab before editing so you can restore it if a configuration error occurs.

    sudo cp /etc/fstab /etc/fstab.bak
  2. Obtain the disk UUID

    Run blkid to obtain the UUID of the target partition. This example uses /dev/vdb1.

    sudo blkid /dev/vdb1

    Sample output. Record the UUID value.

    /dev/vdb1: UUID="f1645951-134f-4677-b5f4-c65c71f8f86d" TYPE="ext4" PARTUUID="..."
  3. Add the mount entry to /etc/fstab

    Edit /etc/fstab and add a mount entry.

    sudo nano /etc/fstab

    Add a line in the following format. Replace <Your-UUID>, <Mount-point>, and <File-system-type> with your actual values.

    # <Device>         <Mount-point>    <File-system-type>    <Mount-options>      <dump> <pass>
    UUID=<Your-UUID>  /mnt        ext4        defaults,nofail   0      2

    About the nofail mount option:

    • Purpose: nofail ensures the instance boots normally even if the disk cannot be mounted.

    • When to use it: Add this option to all non-critical data disks.

    • Risk note: The system does not report mount failures during boot. Monitor the mount point in later steps to detect issues.

  4. Verify the automatic mount configuration

    Test the /etc/fstab configuration without restarting.

    # Unmount the current mount point, for example /mnt
    sudo umount /mnt
    
    # Reload all entries in /etc/fstab
    sudo mount -a
    
    # Check whether the mount succeeded
    sudo lsblk -f

    If lsblk -f shows the device mounted to the correct directory, the configuration succeeds. If mount -a reports an error, the /etc/fstab configuration has a problem. Connect to the instance using VNC and run sudo mv /etc/fstab.bak /etc/fstab to restore the backup.

Configure automatic process startup and keepalive at boot

Register your business process as a systemd service for automatic startup at boot and automatic restart on unexpected exit.

  • Method 1: systemd (recommended): Provides fine-grained control over service dependencies, run-as users, and environment variables. Best practice for production environments.

  • Method 2: Use the Cloud Assistant plug-in for service keep-alive: The Cloud Assistant ecs-tool-servicekeepalive plug-in auto-generates systemd configuration files. Suitable for basic scenarios without detailed service management needs.

This tutorial uses manual systemd configuration (Method 1).

  1. (Optional but recommended) Create a dedicated user for the application

    Follow the principle of least privilege. Avoid running business processes as root.

    sudo groupadd myapp
    sudo useradd -r -s /bin/false -g myapp myapp
  2. Create a systemd service configuration file

    Create a .service file in /etc/systemd/system/, such as myapp.service.

    sudo nano /etc/systemd/system/myapp.service

    Paste the following template and update it for your environment.

    [Unit]
    Description=My Application Service
    # Start the service after network and remote file systems are ready
    After=network-online.target remote-fs.target
    Wants=network-online.target
    # Uncomment the next line if the service depends on the disk mounted in Step 1 (for example, /mnt)
    # RequiresMountsFor=/mnt
    
    [Service]
    # --- Security configuration ---
    # Run the service as the dedicated user and group
    User=myapp
    Group=myapp
    
    # --- Runtime configuration ---
    # Working directory for the process
    WorkingDirectory=/opt/myapp
    # Absolute path and arguments for the start command
    ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.json
    
    # --- Automatic restart policy ---
    # Restart only if the service exits abnormally (for example, crashes)
    Restart=on-failure
    # Time interval between restart attempts
    RestartSec=5s
    
    [Install]
    # Define the target for enabling the service. multi-user.target means enable in multi-user mode.
    WantedBy=multi-user.target
  3. Enable and start the service

    Reload the systemd configuration, enable boot-time auto-start, and start the service:

    # Reload systemd configuration
    sudo systemctl daemon-reload
    # Enable boot-time autostart
    sudo systemctl enable myapp.service
    # Start the service now
    sudo systemctl start myapp.service
  4. Check the service status

    Verify the service is running and enabled for auto-start.

    sudo systemctl status myapp.service

    If Active shows active (running) and Loaded includes enabled, the configuration succeeds.

Configure monitoring and alerting

Set up monitoring to receive alerts when automatic recovery fails.

  1. Mount point availability monitoring

    • Purpose: Ensure key mount points (such as /mnt) stay available.

    • How to implement: Use a script (such as df -h) to check mount points regularly and alert if a mount is missing.

  2. Key process liveness monitoring

    • Purpose: Ensure core processes (such as myapp) are running.

    • How to implement: Use Cloud Monitor’s process monitoring feature.

  3. Instance unexpected restart monitoring

See Cloud Monitor for detailed monitoring configuration.

Validate configuration persistence (crash drill)

Simulate a failure to test the full self-healing and alerting system.

Important

Fault injection causes instance restarts and service interruptions. Run this drill only in non-production environments or during approved maintenance windows.

  1. Inject the fault

    Use the Cloud Assistant plug-in ecs-fault-oscrash to simulate a kernel panic and trigger an unexpected restart :

    sudo acs-plugin-manager --exec --plugin ecs-fault-oscrash --params inject
  2. Wait for the instance to restart

    The instance crashes and restarts immediately. Wait three to five minutes, then reconnect.

  3. End-to-end validation

    • Check disk mounting: Run lsblk -f and confirm the data disk mounted automatically.

    • Check the business process: Run systemctl status myapp.service and confirm the service shows active (running).

    • Check monitoring and alerting: Log in to the Cloud Monitor console and verify that event alerts were triggered.