How to use Instance Health Diagnostics to fix an instance that fails to start-阿里云帮助中心

If high inode or space usage on a disk prevents a Linux instance from starting, use Instance Health Diagnostics to resolve the issue.

Prerequisites

Instance Health Diagnostics has detected high inode or space usage on the ECS instance's disk.

The ECS instance and its resources meet the following conditions:

Resource	Conditions
Instance	The instance is an I/O optimized instance. The following instance types are not supported: ecs.ebmc4.8xlarge, ecs.ebmhfg5.2xlarge, and ecs.ebmg5.24xlarge.
Disk	The disk is in the In Use state. The disk is an enterprise SSD, standard SSD, or ultra disk. After you renew a subscription ECS instance and downgrade its configuration, you cannot resize its disks for the rest of the billing cycle. The new disk capacity cannot exceed the maximum capacity for its category. For more information, see Limits of Elastic Block Storage. Important A disk with an MBR partition cannot be resized to 2 TiB or larger. If you must resize an MBR partition beyond 2 TiB, create a new disk larger than 2 TiB, format it with a GPT partition, and then copy the data from the old MBR partition to the new GPT partition. For instructions on how to format a GPT partition, see Partition and format a data disk larger than 2 TiB.

Background

Problem description: High inode or space usage on a disk is preventing a Linux instance from starting.

Cause: The instance's disk has insufficient capacity and requires resizing.

Note

You can run the df -Th command to check the disk space usage and run the df -i command to check the disk inode usage.

You can resize a cloud disk by using a repair disk from Instance Health Diagnostics or by using an intermediate instance. For more information, see Resize a cloud disk by using a repair disk or Resize a cloud disk by using an intermediate instance.

Resize a disk with a repair disk

Instance Health Diagnostics allows you to resize a disk by using a repair disk.

In the console, create a snapshot of the target disk. For more information, see Create a snapshot.
In the console, resize the target disk. For more information, see Expand the capacity of a disk.
Establish a remote connection to the affected instance.
When an ECS instance is in repair mode with a mounted repair disk, you can connect to the instance only by using VNC. For more information, see Connect to an instance by using VNC.

Run the following command to view disk information for the instance.

fdisk -lu

In the output, find device names in the /dev/vd* format, such as /dev/vda, /dev/vdb, and /dev/vdc. These devices are the disks of the affected instance.

The following example shows three partitions: a system disk (/dev/vda1) and two data disks (/dev/vdb1 and /dev/vdc1). The command output is as follows:

[root@ecs ~]# fdisk -lu
Disk /dev/vda: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b83c7
   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048    83886046    41941999+  83  Linux
Disk /dev/vdb: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xae44111b
   Device Boot      Start         End      Blocks   Id  System
/dev/vdb1            2048    83886079    41942016   83  Linux
Disk /dev/vdc: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000
   Device Boot      Start         End      Blocks   Id  System
/dev/vdc1               1    83886079    41943039+  ee  GPT
[root@ecs ~]#

No.	Partition	Description
①	`/dev/vda1`	System disk. A System value of Linux indicates an MBR partition.
②	`/dev/vdb1`	Data disk. A System value of Linux indicates an MBR partition.
③	`/dev/vdc1`	Data disk. A System value of GPT indicates a GPT partition.

Note

If the query result shows that the disk capacity is still 40 GiB (Disk /dev/vda: 42.9 GB), it means the resize was unsuccessful. We recommend that you restart the instance in the console.

Run the following command to identify the file system type for each partition.

df -Th

[root@ecs ~]# df -Th
Filesystem     Type      Size  Used Avail Use% Mounted on
devtmpfs       devtmpfs  1.8G     0  1.8G   0% /dev
tmpfs          tmpfs     1.8G     0  1.8G   0% /dev/shm
tmpfs          tmpfs     1.8G  612K  1.8G   1% /run
tmpfs          tmpfs     1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/vda1      ext4       40G  1.8G   36G   5% /
tmpfs          tmpfs     361M     0  361M   0% /run/user/0
/dev/vdb1      ext4       40G   49M   38G   1% /media/vdb
/dev/vdc1      xfs        40G   74M   40G   1% /media/vdc
[root@ecs ~]#

Run the following command to resize the partition.
```
growpart /dev/vda 1
```
This example resizes a system disk partition. A space is required between /dev/vda and 1. Adapt the command for your actual partition.
```
[root@ecs ~]# growpart /dev/vda 1
CHANGED: partition=1 start=2048 old: size=83883999 end=83886047 new: size=125827039 end=125829087
```
On the ECS instance, resize the file system based on its type.
- Resize an ext* file system (such as ext4): Run the following commands to resize the file system.
  Resize the file system of the system disk /dev/vda1.
```
resize2fs /dev/vda1    
```
  Resize the file system of the data disk /dev/vdb1.
```
resize2fs /dev/vdb1          
```
  Note
  /dev/vda1 and /dev/vdb1 are partition names. Replace them with your actual partition names.
- Resize an xfs file system: Run the following command to resize the file system of the data disk /dev/vdc1.
```
xfs_growfs /media/vdc
```
  Note
  /media/vdc is the mount point of /dev/vdc1. Replace it with your actual mount point.
- Resize a btrfs file system: If the data disk uses a btrfs file system, run the following command to resize the file system.
```
btrfs filesystem resize max /mountpoint
```
  Note
  /mountpoint is the mount point of the data disk. Replace it with your actual mount point.

Run the following command to verify the resize.

df -Th

The following is an example output:

In this example, /dev/vda1 (ext4, 59G, mounted on /), /dev/vdb1 (ext4, 59G, mounted on /media/vdb), and /dev/vdc1 (xfs, 60G, mounted on /media/vdc) show the resized partition capacities.

[root@ecs ~]# df -Th
Filesystem      Type      Size  Used Avail Use% Mounted on
devtmpfs        devtmpfs  1.8G     0  1.8G   0% /dev
tmpfs           tmpfs     1.8G     0  1.8G   0% /dev/shm
tmpfs           tmpfs     1.8G  492K  1.8G   1% /run
tmpfs           tmpfs     1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/vda1       ext4       59G  1.9G   55G   4% /
tmpfs           tmpfs     361M     0  361M   0% /run/user/0
/dev/vdb1       ext4       59G   52M   56G   1% /media/vdb
/dev/vdc1       xfs        60G   94M   60G   1% /media/vdc

After resizing, verify that your data is intact.

If the resize is successful and your applications run as expected, no further action is needed.
If the resize fails, restore the disk from the snapshot you created.

After the repair is complete, exit the repair environment and then check the status of the problematic instance.
1. On the Instance Health Diagnostics page of the ECS console, detach the repair disk and restore the instance to normal mode.
2. Connect to the repaired ECS instance and verify the disk capacity.

Resize a disk with an intermediate instance

Alternatively, you can detach the problematic disk, attach it to an intermediate instance (another Linux instance in the same region), resize the disk, and then reattach it to the original ECS instance.

Detach the disk from the problematic instance. For more information, see Detach a system disk or Detach a data disk.
Attach the disk to the intermediate instance. For more information, see Attach a data disk.
On the intermediate instance, resize the disk. For more information, see Expand the capacity of a disk.
After resizing, detach the disk from the intermediate instance. For more information, see Detach a data disk.
Reattach the disk to the original ECS instance. For more information, see Detach or attach a system disk or Attach a data disk.
Connect to the repaired ECS instance and verify the disk capacity.