Skip to main content

Disks, logs and certificates

What it is​

Three slow failures: disks fill up with logs and old images, logs are kept forever and cost money, and certificates expire. None of them breaks anything until the day it breaks everything.

Why we do it​

  • A client's application log grew by about 1.5 GB a day.
  • Several log groups had no retention, so they were kept forever.
  • Our self-hosted Compose files had no log size limits.
  • Certificate renewal on VPS boxes had no expiry monitoring.

Steps​

1. Disks​

WhereControl
Docker logslogging.options.max-size and max-file in every Compose service
Old imagesdocker image prune -f after each deploy; docker system prune monthly, reviewed first
App log fileslogrotate daily, keep 7 days, compress
UploadsNever on the server's disk. Use S3
AlarmDisk above 80% on every self-hosted box
df -h /                               # how full
sudo du -xh / --max-depth=2 | sort -h | tail -20   # what is big
docker system df                       # what Docker is using

2. Log retention​

LogsKeep
Application logs (CloudWatch)14 days in dev, 30 in prod, unless the client needs more
Load balancer access logs (S3)90 days
CloudTrail365 days
Database logs14 days
Lambda and CodeBuild logs14 days

Set retention_in_days on every log group in Terraform. A log group created by AWS automatically (Lambda, RDS, CodeBuild) has no retention until you set it.

# find log groups that keep data forever
aws logs describe-log-groups --query 'logGroups[?!retentionInDays].logGroupName' --output text

3. Certificates​

WhereRenews byYou check
AWS (ACM)Automatically, if the DNS validation record existsThe record is still in Route 53
Traefik, CaddyAutomaticallyThe ACME storage volume persists across restarts
nginx with CertbotThe Certbot timer`systemctl list-timers
Client-provided certificatesA personA calendar reminder 30 days before expiry

Add an expiry check for every public hostname, so a broken renewal is caught weeks early:

echo | openssl s_client -servername api.example.com -connect api.example.com:443 2>/dev/null \
  | openssl x509 -noout -enddate

A monitoring service or a small scheduled job can run this daily and alert under 21 days.

Rules​

Rule

Every log group, in every account, has a retention period. "Forever" is never the default.

Rule

Every public certificate has an expiry alert at 21 days, whichever tool renews it.

Done when​

  • Log limits on every container; logrotate for log files; disk alarm at 80%.
  • Retention set on every log group, checked with the query above.
  • Each certificate's renewal method known; expiry alerts at 21 days.
Last updated on