Skip to main content

Server mistakes

Every item here happened on a server we ran or were responsible for in 2026. Read it in your first week, and add to it.

The pattern behind most of them​

Access and building​

  • A build on a 2 GB server ran out of memory and killed the management agent. The box was unreachable for 25 minutes until a reboot. Build in CI.
  • A server's git remote held an expired token, so git pull deploys broke. Ship images, not checkouts.
  • A password manager server went down and staff lost their TOTP codes. Critical tools need monitoring and a backup path.
  • Server passwords and private keys were kept in text files in a shared folder.
  • Wrong builds were deployed by hand to a demo server, and one service ran a build about three months old.

Backups​

  • An empty 20-byte dump passed every check for weeks.
  • A dev box had no backups at all.
  • Dumps went to /tmp on the same server.
  • A client's backups sat on the compromised database server.
  • Manual production snapshots never expired.

Upgrades and patching​

  • A retired download URL broke deploys for three weeks, silently.
  • A full OS update ran out of memory on a tiny instance.
  • Web apps ran a Node version past end of life.
  • Terraform defaulted to a database version AWS no longer creates.

Disks and logs​

  • Compose services had no log limits.
  • Log groups kept data forever.
  • A client's app log grew 1.5 GB a day and recorded neither paths nor client addresses.

Cost and clean-up​

  • Stacks ran for weeks with nothing pointing at them.
  • Load-test clones kept costing after the test.
  • A retired stack left an admin IAM group and hand-made parameters behind.
  • A dev box ran past its delete-after date.

Monitoring​

  • No uptime check on any self-hosted box. Each outage was found by a person.
  • Alarm noise led people to ignore alerts, until alarms were split into notify and silent tiers.
Last updated on