Linux Is Out of Disk Space: A Systematic Way to Find Out Why

"No space left on device" has several distinct causes that look identical from the error message. Checking them in the right order finds the problem in minutes instead of an afternoon, and avoids the common trap of deleting something that was not the cause.

Step 1: is it actually blocks, or inodes?

`` df -h # block usage df -i # inode usage ``

A filesystem can be 20% full by size and completely out of inodes — one per file, allocated at filesystem creation. Millions of tiny files (session data, mail queues, cache entries) exhaust inodes long before space. If df -i shows 100%, deleting large files will not help; you need to delete *many* files.

The arithmetic is fixed at mkfs time. ext4 allocates one inode per *bytes-per-inode* ratio, commonly 16 KiB for a general-purpose volume. A 100 GiB filesystem built that way gets 107,374,182,400 ÷ 16,384 = 6,553,600 inodes. That sounds like plenty until a build cache or a Maildir produces four million files averaging 2 KiB each: those files occupy only about 7.6 GiB of the 100 GiB, yet consume 61% of the inodes. Change the ratio and you change the ceiling — mkfs.ext4 -i 8192 doubles the inode count and halves the bytes available for data.

XFS allocates inodes dynamically, so it does not fail this way, which is one practical reason to prefer it for workloads that create vast numbers of small files. Find the offenders with:

`` du --inodes -x --max-depth=2 /var | sort -n | tail -20 ``

Step 2: find the large directories

`` du -xh --max-depth=1 / | sort -h | tail -20 ``

The -x matters: it stops du crossing filesystem boundaries, so you are not shown the contents of network mounts while investigating the root disk.

Then descend into whichever directory is largest, repeating.

Step 3: when df and du disagree

If df reports a full disk but du cannot account for the space, the usual cause is a deleted file still held open by a process. Unlinking a file only removes the directory entry; the blocks are freed when the last file descriptor closes.

`` lsof +L1 ``

This lists open files with a link count below 1 — deleted but still held. The classic case is a log file removed by hand while the service still has it open, often gigabytes.

Fix by restarting the holding process, or truncate in place instead:

`` : > /var/log/offending.log ``

Truncating keeps the descriptor valid, so the writing process is unaffected — which is why it is preferable to rm for active logs.

Step 4: check the usual suspects

• /var/log — a runaway log with broken rotation
• /var/lib/docker — images, stopped containers, dangling volumes
• /var/cache — package manager caches
• /tmp — sometimes never cleaned
• Old kernels in /boot, which is often a small partition
• Core dumps, frequently in /var/lib/systemd/coredump

Step 5: reserved blocks

ext4 reserves 5% for root by default. A filesystem "100% full" for normal users may still have space root can use — which is why a service fails while your shell commands still work. On a 500 GB volume that reservation is 25 GB held back; dropping it to 1% releases 20 GB and still leaves 5 GB for root to recover with. Check and adjust:

`` tune2fs -l /dev/sda1 | grep -i reserved tune2fs -m 1 /dev/sda1 ``

On a large modern disk, 5% is a lot to hold back; 1% is usually plenty. The change takes effect immediately and needs no unmount.

Step 6: remember the capacity you never had

Two separate deductions mean a "1 TB" disk never shows 1 TB free:

• Unit mismatch. Drives are sold in decimal GB. 10¹² bytes is

10¹² ÷ 1024⁴ = 0.91 TiB, so df reports about 931 GiB for a 1 TB disk before any filesystem exists. That is 9% "missing" and entirely expected.

• Filesystem metadata. Inode tables, journal and group descriptors consume

a further slice at format time.

Quantify both with the filesystem overhead calculator before concluding that something is wrong.

When you need space right now

In order of safety, with the least destructive first:

1. Truncate the largest active log in place (: > file).
2. Clear the package cache — apt clean or dnf clean all.
3. Drop the reserved-block percentage to buy minutes.
4. journalctl --vacuum-size=200M to cap systemd journals.
5. docker system prune -a --volumes only once you are certain nothing

stopped is needed; this one is not reversible.

Never rm -rf a directory you have not inspected, and never delete anything under /var/lib belonging to a database to free space — the recovery is far more expensive than the outage.

Preventing recurrence

• Configure logrotate with both size and time limits
• Set container log limits (max-size, max-file)
• Monitor at 80%, not 95% — you need time to react
• Watch inode usage alongside block usage
• Schedule docker system prune if you build on the host
• Alert on rate of growth, not just the level: a volume that gained 20% in an

hour is a problem even at 40% full

Size new volumes with the filesystem utilization calculator, check the inode ceiling for your expected file count with the inode usage calculator, and remember that filesystem overhead means usable capacity is always below the nominal figure.