First published 2026-07
Based on prior experience and the logs entries like
{"ip":"216.73.216.95", "user":"-", "time":"24/Jun/2026:01:22:13 +0000", "req":"GET /review_derivation/0000000011 HTTP/1.1", "stat":"200", "bsnt":"85008", "reqt":"2.647", "ref":"-", "ua":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)", "gz":"-"}
{"ip":"17.246.23.224", "user":"-", "time":"24/Jun/2026:01:22:14 +0000", "req":"GET /login?next=https://derivationmap.net/edit_step/207210/3408108 HTTP/1.1", "stat":"302", "bsnt":"685", "reqt":"0.192", "ref":"-", "ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)", "gz":"-"}
{"ip":"85.208.96.211", "user":"-", "time":"24/Jun/2026:01:22:18 +0000", "req":"GET /new_step_expressions/0000000003/0000111732 HTTP/1.1", "stat":"200", "bsnt":"373203", "reqt":"0.529", "ref":"-", "ua":"Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)", "gz":"-"}
{"ip":"85.208.96.210", "user":"-", "time":"24/Jun/2026:01:22:12 +0000", "req":"GET /new_step_expressions/0000332170/0000111246 HTTP/1.1", "stat":"200", "bsnt":"373263", "reqt":"1.038", "ref":"-", "ua":"Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)", "gz":"-"}
{"ip":"145.239.69.153", "user":"-", "time":"24/Jun/2026:01:22:12 +0000", "req":"GET /blog/page/2026/03/DigitalOcean%20initial%20configuration HTTP/1.1", "stat":"200", "bsnt":"25859", "reqt":"0.083", "ref":"-", "ua":"Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)", "gz":"-"}
I expected bots were overwhelming the CPU. I looked at top
Gemini 3.5 Pro suggested trying `sudo tail -f /var/log/syslog` which showed
2026-06-24T01:28:53.177943+00:00 ubuntu-4gb-hel1-4 rsyslogd[2325666]: rsyslogd: file '/var/log/auth.log'[9] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2312.0 try https://www.rsyslog.com/e/2027 ]
$ sudo du -hxd 1 / | sort -hr 37G / 32G /var 2.7G /home 1.8G /usr 116M /boot 41M /opt 11M /root
and then
$ sudo du -hxd 1 /var | sort -hr 32G /var 31G /var/lib 883M /var/log 114M /var/cache 1.9M /var/backups 332K /var/crash
and then
$ sudo du -hxd 1 /var/lib | sort -hr 31G /var/lib/containerd 31G /var/lib 177M /var/lib/apt 28M /var/lib/docker 23M /var/lib/dpkg
and then
$ sudo du -hxd 1 /var/lib/containerd | sort -hr 31G /var/lib/containerd 29G /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs 1.9G /var/lib/containerd/io.containerd.content.v1.content 3.0M /var/lib/containerd/io.containerd.metadata.v1.bolt
sudo docker system prune -a --volumes
which resulted in
Total reclaimed space: 5.802GB
That's an improvement, but
$ sudo du -hxd 1 /var/lib/containerd | sort -hr 27G /var/lib/containerd 25G /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs 1.8G /var/lib/containerd/io.containerd.content.v1.content 3.0M /var/lib/containerd/io.containerd.metadata.v1.bolt
$ docker images i Info → U In Use IMAGE ID DISK USAGE CONTENT SIZE EXTRA neo4j:4.4.45-community 4322925e2e86 892MB 330MB U ui_v8-flask-webserver:latest c847dcc46538 6.01GB 1.56GB U
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0d53d4dab9e7 ui_v8-flask-webserver "gunicorn --reload -…" 7 weeks ago Up 33 hours 5000/tcp ui_v8-flask-webserver-1 f3646f991c05 neo4j:4.4.45-community "tini -g -- /startup…" 7 weeks ago Up 33 hours (healthy) 0.0.0.0:7473-7474->7473-7474/tcp, [::]:7473-7474->7473-7474/tcp, 0.0.0.0:7687->7687/tcp, [::]:7687->7687/tcp ui_v8-neo4j_docker-1
This issue occurs because frequent builds using docker build and docker compose up --build generate dangling images (intermediate layers that are no longer associated with a tagged image) and accumulate build cache.
While docker compose down --volumes successfully removes the containers, networks, and volumes defined in your compose file, it does not clean up the unused image layers or the build cache stored by Docker and its container runtime (containerd). Over time, these layers accumulate in /var/lib/containerd/.../overlayfs.
Here is how you can safely clean up this disk space and prevent it from filling up again.
Modern versions of Docker use BuildKit, which caches build steps to make subsequent builds faster. This cache can grow quite large over several months.
To remove the build cache, run:
docker builder prune
To remove all build cache, including the cache for unused images:
docker builder prune -a
Each time you rebuild ui_v8_flask_webserver:latest-amd64, the old version of that image loses its tag and becomes a "dangling" image.
To remove only dangling images (those listed as <none>:<none>):
docker image prune
If you want to remove all images that are not currently associated with a running container (including older versions of other images you might have pulled):
docker image prune -a
Note: This will require Docker to re-download or rebuild images the next time you run them if they aren't currently running.
If you want to clean up containers, networks, dangling images, and build cache all at once, you can run:
docker system prune
If you want to be more aggressive and remove all unused images (not just dangling ones) and the build cache, add the -a flag:
docker system prune -a
To prevent your disk from filling up again, you can periodically run a cleanup command after stopping your containers. For example:
# 1. Stop and remove containers/volumes
docker compose down --volumes --remove-orphans
# 2. Clean up dangling images and build cache
docker system prune -f
Adding docker system prune -f (the -f flag bypasses the confirmation prompt) to your development or deployment scripts once a week or once a month is usually sufficient to manage disk space.
$ docker builder prune WARNING! This will remove all dangling build cache. Are you sure you want to continue? [y/N] y ID RECLAIMABLE SIZE LAST ACCESSED f45scwx1wsdjhw4qdqzrcpw00* true 12.29kB 39 seconds ago i587a0dgc4pfvn1b1bve06cwy* true 23.28MB 39 seconds ago n67s18ay8o6i0kd6iac3v9go4* true 4.096kB 39 seconds ago Total: 23.3MB
$ docker builder prune -a WARNING! This will remove all build cache. Are you sure you want to continue? [y/N] y ID RECLAIMABLE SIZE LAST ACCESSED 441zs5ehw7knuozrey9jhll3y true 36.78MB About a minute ago 8zihbjs7jjmqhu43vbmbv5hc1 true 461.6MB About a minute ago brjy0s8b9p4naa6vt90merq47 true 1.788GB About a minute ago ... Total: 5.817GB