Persist startup configurations for ECS instances
Configure persistent disk mounts, process auto-start, and monitoring for ECS instances to survive unexpected restarts.
Use cases
ECS instances may restart unexpectedly due to OOM errors or host failures. Temporary configurations—such as manually mounted disks or manually started processes—are lost after a restart, causing data disk unavailability and extended downtime.
Solution architecture
This solution converts temporary operations into permanent settings so the instance automatically restores its business state after a restart. It combines OS-native features with Alibaba Cloud monitoring to enable automatic recovery and alerting.
-
Persistent disk mounting: Use
/etc/fstabto record disk mount information with the device UUID instead of the device name (such as/dev/vdb). This avoids mount failures caused by device name changes after a restart. -
Persistent business process management: Use
systemdto manage business processes as system services.systemdstarts services at boot and restarts them on unexpected exit. -
Status monitoring and validation:
-
Monitoring: Configure Cloud Monitor to track disk mount points, core processes, and instance restart events. Receive alerts when automatic recovery fails.
-
Validation: Use the Cloud Assistant fault injection plug-in to simulate crashes in non-production environments and verify that configuration persistence and alerting work as expected.
-
Procedure
Complete disk mounting, process auto-start, monitoring, and validation in the following steps.
Configure automatic disk mounting at boot
Use the disk UUID in /dev/vdb1's fstab entry to ensure automatic mounting after restarts. UUIDs remain constant across the instance lifetime, unlike device names.
-
Back up the fstab file
Back up
/etc/fstabbefore editing so you can restore it if a configuration error occurs.sudo cp /etc/fstab /etc/fstab.bak -
Obtain the disk UUID
Run
blkidto obtain the UUID of the target partition. This example uses/dev/vdb1.sudo blkid /dev/vdb1Sample output. Record the
UUIDvalue./dev/vdb1: UUID="f1645951-134f-4677-b5f4-c65c71f8f86d" TYPE="ext4" PARTUUID="..." -
Add the mount entry to /etc/fstab
Edit
/etc/fstaband add a mount entry.sudo nano /etc/fstabAdd a line in the following format. Replace
<Your-UUID>,<Mount-point>, and<File-system-type>with your actual values.# <Device> <Mount-point> <File-system-type> <Mount-options> <dump> <pass> UUID=<Your-UUID> /mnt ext4 defaults,nofail 0 2About the
nofailmount option:-
Purpose:
nofailensures the instance boots normally even if the disk cannot be mounted. -
When to use it: Add this option to all non-critical data disks.
-
Risk note: The system does not report mount failures during boot. Monitor the mount point in later steps to detect issues.
-
-
Verify the automatic mount configuration
Test the
/etc/fstabconfiguration without restarting.# Unmount the current mount point, for example /mnt sudo umount /mnt # Reload all entries in /etc/fstab sudo mount -a # Check whether the mount succeeded sudo lsblk -fIf
lsblk -fshows the device mounted to the correct directory, the configuration succeeds. Ifmount -areports an error, the/etc/fstabconfiguration has a problem. Connect to the instance using VNC and runsudo mv /etc/fstab.bak /etc/fstabto restore the backup.
Configure automatic process startup and keepalive at boot
Register your business process as a systemd service for automatic startup at boot and automatic restart on unexpected exit.
-
Method 1: systemd (recommended): Provides fine-grained control over service dependencies, run-as users, and environment variables. Best practice for production environments.
-
Method 2: Use the Cloud Assistant plug-in for service keep-alive: The
Cloud Assistantecs-tool-servicekeepaliveplug-in auto-generatessystemdconfiguration files. Suitable for basic scenarios without detailed service management needs.
This tutorial uses manual systemd configuration (Method 1).
-
(Optional but recommended) Create a dedicated user for the application
Follow the principle of least privilege. Avoid running business processes as
root.sudo groupadd myapp sudo useradd -r -s /bin/false -g myapp myapp -
Create a systemd service configuration file
Create a
.servicefile in/etc/systemd/system/, such asmyapp.service.sudo nano /etc/systemd/system/myapp.servicePaste the following template and update it for your environment.
[Unit] Description=My Application Service # Start the service after network and remote file systems are ready After=network-online.target remote-fs.target Wants=network-online.target # Uncomment the next line if the service depends on the disk mounted in Step 1 (for example, /mnt) # RequiresMountsFor=/mnt [Service] # --- Security configuration --- # Run the service as the dedicated user and group User=myapp Group=myapp # --- Runtime configuration --- # Working directory for the process WorkingDirectory=/opt/myapp # Absolute path and arguments for the start command ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.json # --- Automatic restart policy --- # Restart only if the service exits abnormally (for example, crashes) Restart=on-failure # Time interval between restart attempts RestartSec=5s [Install] # Define the target for enabling the service. multi-user.target means enable in multi-user mode. WantedBy=multi-user.target -
Enable and start the service
Reload the
systemdconfiguration, enable boot-time auto-start, and start the service:# Reload systemd configuration sudo systemctl daemon-reload # Enable boot-time autostart sudo systemctl enable myapp.service # Start the service now sudo systemctl start myapp.service -
Check the service status
Verify the service is running and enabled for auto-start.
sudo systemctl status myapp.serviceIf
Activeshowsactive (running)andLoadedincludesenabled, the configuration succeeds.
Configure monitoring and alerting
Set up monitoring to receive alerts when automatic recovery fails.
-
Mount point availability monitoring
-
Purpose: Ensure key mount points (such as
/mnt) stay available. -
How to implement: Use a script (such as
df -h) to check mount points regularly and alert if a mount is missing.
-
-
Key process liveness monitoring
-
Purpose: Ensure core processes (such as
myapp) are running. -
How to implement: Use Cloud Monitor’s process monitoring feature.
-
-
Instance unexpected restart monitoring
-
Purpose: Detect system events.
-
How to implement: Use Cloud Monitor’s system event alerting feature.
-
See Cloud Monitor for detailed monitoring configuration.
Validate configuration persistence (crash drill)
Simulate a failure to test the full self-healing and alerting system.
Fault injection causes instance restarts and service interruptions. Run this drill only in non-production environments or during approved maintenance windows.
-
Inject the fault
Use the Cloud Assistant plug-in
ecs-fault-oscrashto simulate a kernel panic and trigger an unexpected restart :sudo acs-plugin-manager --exec --plugin ecs-fault-oscrash --params inject -
Wait for the instance to restart
The instance crashes and restarts immediately. Wait three to five minutes, then reconnect.
-
End-to-end validation
-
Check disk mounting: Run
lsblk -fand confirm the data disk mounted automatically. -
Check the business process: Run
systemctl status myapp.serviceand confirm the service showsactive (running). -
Check monitoring and alerting: Log in to the Cloud Monitor console and verify that event alerts were triggered.
-