What do I do if no disk space is available or the number of inodes exceeds the inode quota in a Linux simple application server?

更新时间:
复制 MD 格式

When creating a file or starting an application on your Linux simple application server triggers a No space left on device error, the root cause isn't always a full disk partition. Run the quick diagnosis below to identify which of the four causes applies, then jump to the corresponding solution.

Note

If disk usage is as expected and you simply need more capacity, upgrade your server, attach a data disk, or extend an existing data disk. See Upgrade a simple application server, Attach a data disk, and Extend the data disk.

Quick diagnosis

Run these two commands to identify the cause before choosing a solution:

df -h   # Check disk space usage per partition
df -i   # Check inode usage per partition
What you observeLikely causeGo to
A partition shows 100% under Use% in df -h outputDisk partition is fullCause 1
A partition shows 100% under IUse% in df -i outputInodes exhaustedCause 2
Both df -h and df -i show normal usage, but the error persistsDeleted files still referenced by processesCause 3
The previous three causes do not explain the issue; du and df report contradictory space usage for the same directoryMount point overwrittenCause 4

Cause 1: Disk partition is full

The space usage of one or more disk partitions has reached 100%. To resolve this, either free up space by deleting unnecessary files or expand disk capacity.

Find and delete large files

  1. Connect to the server using the rescue feature. See Connect to a Linux server by using the rescue feature.

  2. Identify which partition is full:

    df -h

    The output shows space usage per partition. Note which partition shows 100% usage.

    image

  3. Find which top-level directory uses the most space:

    cd /
    du -sh *

    The output lists disk usage for each directory under /. In the following example, /usr uses the most space.

    image

  4. Drill down into that directory and repeat until you find the specific file or directory consuming the space. For example, if /usr uses the most space:

    cd /usr
    du -sh *

    Keep repeating this pattern — navigate into the largest directory and run du -sh * — until you identify the specific culprit.

    image

  5. Delete files or directories that are no longer needed.

Expand disk capacity

If you cannot free enough space by deleting files, expand disk capacity instead. See Upgrade a simple application server, Attach a data disk, or Extend the data disk.

Cause 2: Inodes exhausted

When inode usage reaches 100%, the OS cannot create new files or directories even if plenty of disk space remains. Every file and directory requires one inode — a metadata record that stores the file type, size, permissions, owner, number of connections to the file, timestamps, and pointers to data blocks.

Note

Only modify inode configuration when inode usage has actually reached 100%.

Check inode usage

  1. Connect to the server using the rescue feature. See Connect to a Linux server by using the rescue feature.

  2. Check inode usage per partition:

    df -i

    If any partition shows 100% (or close to it) under IUse%, proceed with one of the following solutions.

    image

Option A: Delete files with high inode usage

This option avoids reformatting. Use it when you can identify and remove unnecessary files.

  1. Count the number of files in each top-level directory:

    for i in /*; do echo $i; find $i | wc -l; done

    The output shows how many files exist under each directory. Inode usage is directly proportional to file count.

    image

  2. Navigate into the directory with the highest file count and repeat the command to drill down further. Delete unnecessary files until inode usage drops to an acceptable level.

Option B: Reformat the partition with more inodes

Use this option when files cannot be deleted or when inode usage remains high after deleting files. Reformatting erases all data on the partition, so back up first.

Warning
  • Formatting deletes all data on the disk. Back up data before proceeding — copy files off the disk or create a snapshot.

  • Unmounting the file system interrupts application services. Schedule this operation during a maintenance window.

  1. Unmount the file system. This example unmounts /home — adjust the mount point to match your setup:

    umount /home
  2. Create a new file system with a larger inode count. This example creates an ext3 file system on /dev/xvdb with 1,638,400 inodes — adjust the device path and inode count for your disk:

    Note

    In most cases, the number of inodes is calculated using the following formula: disk capacity (KB) / 16 KB. For example, a 40 GB disk may have approximately 2,621,440 inodes based on this formula. The maximum is 2^32 (approximately 4.3 billion). Specify an appropriate number of inodes based on the disk capacity.

    mkfs.ext3 /dev/xvdb -N 1638400
  3. Mount the new file system based on your /etc/fstab configuration:

    mount -a
  4. (Optional) Verify that the inode count increased:

    dumpe2fs -h /dev/xvdb | grep node

    After confirming the increased inode count, copy your backup data back to the disk and restart affected applications.

    image

Cause 3: Deleted files still referenced by processes

If df -h and df -i both show normal usage but the error persists, deleted files held open by running processes are likely consuming the hidden space.

When a process deletes a file but keeps a file handle open, Linux marks the file as deleted but cannot release its disk space until the handle is closed. This space is invisible to both df and du.

Note

This is why df and du sometimes report different disk usage numbers — df counts space allocated to the partition including space held by deleted-but-open files, while du counts only space reachable through the directory tree.

Find and release held file handles

  1. Connect to the server using the rescue feature. See Connect to a Linux server by using the rescue feature.

  2. If lsof is not installed, install it:

    • Alibaba Cloud Linux or CentOS:

      yum install -y lsof
    • Debian or Ubuntu:

      apt-get install -y lsof
  3. List deleted files that are still held open, sorted by size:

    lsof |grep delete | sort -k7 -rn | more

    File sizes appear in the seventh column. If the sum of these sizes is close to the amount of unexplained disk usage, deleted files are the cause.

    image

  4. Release the file handles using one of the following methods:

    Restart the instance

    Restarting terminates all running processes and releases their file handles, freeing the disk space.

    Important

    Instance restarts interrupt running services. Schedule restarts during a maintenance window.

    Kill the process holding the file

    If restarting the instance is not an option, terminate only the process that holds the file handle.

    1. List processes with open deleted files:

      lsof |grep delete

      The process ID (PID) is often displayed in the second column.

    2. Terminate the process:

      kill <PID>
    Important

    Terminating a process may affect services that depend on it. Verify which service the process belongs to before proceeding.

Cause 4: Mount point overwritten

If the previous three causes do not explain the issue, a mount point may have been overwritten.

This happens when a file system is mounted on a directory that already contains data. The new file system hides the original data — writes go to the new file system, but the original data still occupies space on the underlying partition. Commands like du only show the new file system's contents, not the hidden data.

Example: /dev/vda1 (system disk, 30 GB) shows 95% usage, and du reports that /home uses 24 GB.

image

After mounting /dev/vdb1 on /home, the system disk /dev/vda1 still shows 95% usage. But now du reports that /home uses only 20 KB — not 24 GB — while /usr still shows more than 1 GB. The 24 GB from /home is now hidden beneath the mounted file system. You cannot identify which directory consumes the largest amount of disk space, which indicates the issue may be caused by overwritten mount points.

image

To confirm and resolve the issue, unmount the partition and check the space usage of the original mount point.

Warning

Unmounting the file system interrupts application services. Schedule this operation during a maintenance window.