Linux Is Out of Disk Space: A Systematic Way to Find Out Why

"No space left on device" has several distinct causes that look identical from the error message. Checking them in the right order finds the problem in minutes instead of an afternoon.

Step 1: is it actually blocks, or inodes?

`` df -h # block usage df -i # inode usage ``

A filesystem can be 20% full by size and completely out of inodes — one per file, allocated at filesystem creation. Millions of tiny files (session data, mail queues, cache entries) exhaust inodes long before space. If df -i shows 100%, deleting large files will not help; you need to delete *many* files.

Step 2: find the large directories

`` du -xh --max-depth=1 / | sort -h | tail -20 ``

The -x matters: it stops du crossing filesystem boundaries, so you are not shown the contents of network mounts while investigating the root disk.

Then descend into whichever directory is largest, repeating.

Step 3: when df and du disagree

If df reports a full disk but du cannot account for the space, the usual cause is a deleted file still held open by a process. Unlinking a file only removes the directory entry; the blocks are freed when the last file descriptor closes.

`` lsof +L1 ``

This lists open files with a link count below 1 — deleted but still held. The classic case is a log file removed by hand while the service still has it open, often gigabytes.

Fix by restarting the holding process, or truncate in place instead:

`` : > /var/log/offending.log ``

Truncating keeps the descriptor valid, so the writing process is unaffected — which is why it is preferable to rm for active logs.

Step 4: check the usual suspects

• /var/log — a runaway log with broken rotation
• /var/lib/docker — images, stopped containers, dangling volumes
• /var/cache — package manager caches
• /tmp — sometimes never cleaned
• Old kernels in /boot, which is often a small partition
• Core dumps, frequently in /var/lib/systemd/coredump

Step 5: reserved blocks

ext4 reserves 5% for root by default. A filesystem "100% full" for normal users may still have space root can use — which is why a service fails while your shell commands still work. Check and adjust:

`` tune2fs -l /dev/sda1 | grep -i reserved tune2fs -m 1 /dev/sda1 ``

On a large modern disk, 5% is a lot to hold back; 1% is usually plenty.

Preventing recurrence

• Configure logrotate with both size and time limits
• Set container log limits (max-size, max-file)
• Monitor at 80%, not 95% — you need time to react
• Watch inode usage alongside block usage
• Schedule docker system prune if you build on the host

Size new volumes with the filesystem utilization calculator, and remember that filesystem overhead means usable capacity is always below the nominal figure — the overhead calculator quantifies how much.