“No space left on device” at 2am means one thing: stop guessing and find the actual culprit before you delete anything. Most disk-full incidents are one of four causes, and you can identify which in under five minutes.

1. Confirm it, and where

df -h

Look at the Use% column. If / is at 100%, that’s your problem. If it’s a separate mount like /var or /var/lib/docker that’s full while / has room, that tells you a lot already — Docker’s storage and logs both commonly live under /var/lib/docker.

2. Find the heavy directories

sudo du -sh /var/* 2>/dev/null | sort -rh | head -10

This sorts the top-level directories under /var by size, largest first. Repeat one level deeper into whatever comes out on top:

sudo du -sh /var/lib/docker/* 2>/dev/null | sort -rh | head -10

Keep drilling until you find the actual large files, not just the large directory.

3. The four usual suspects

Docker images and containers. Old, unused images pile up fast, especially with CI building a new tagged image on every push. Check:

docker system df

This breaks down space used by images, containers, volumes, and build cache separately — useful because the fix differs (see the note on pruning below; don’t just run a broad prune blind).

Log files that were never rotated. A chatty app or a container without log rotation configured can write gigabytes to /var/lib/docker/containers/*/*.json.log or /var/log. Find the worst offenders:

sudo find /var/log -type f -size +100M -exec ls -lh {} \;

Old kernel versions. Ubuntu/Debian keep multiple kernel versions around after updates, and they add up over months of patching:

dpkg --list | grep linux-image

Removing old kernels (sudo apt autoremove --purge) is generally safe — it won’t touch the currently running kernel — but confirm uname -r matches the one you’re keeping before you run it.

Database growth, WAL/bloat. Postgres write-ahead logs or an un-vacuumed table can quietly eat disk over weeks. Check your data directory size directly if you suspect this — it’s a slower burn than the other three, so it’s less likely to be the sudden cause, but worth ruling out if nothing else explains it.

4. Fix, in order of safety

Start with what’s unambiguously safe to remove, then escalate:

1. Dangling Docker images — layers that are no longer tagged or referenced by any container: docker image prune (not -a, which also removes unused-but-tagged images you might want).
2. Stopped containers you’re not going to restart: docker container prune.
3. Build cache, which can be surprisingly large on a server that’s built a lot of images: docker builder prune.
4. Old kernels, as above.
5. Log files — truncate rather than delete a file a running process has open, or the process keeps writing to the now-unlinked file and you don’t actually reclaim space until it’s restarted: sudo truncate -s 0 /var/log/some-huge.log.

What not to do

Don’t run docker system prune -a --volumes as a first move. It’s the fastest way to reclaim space and also the fastest way to delete a volume backing a database you needed. Prune specific categories once you know what’s actually large, not everything at once because you’re in a hurry.

Don’t delete files under /var/lib/docker by hand — the Docker daemon’s internal bookkeeping doesn’t expect files removed outside its own commands, and you can end up with a daemon that won’t start.

The takeaway

df -h tells you it’s full. du -sh sorted and drilled down tells you why. Spend the ninety seconds finding the actual cause before deleting anything — a disk-full incident fixed by guessing is usually a data-loss incident with extra steps.