Mistakes we made

Every DevOps mistake from this module on one page. Each one cost real time, an outage, or a security exposure. Read it before your first production release, then follow the link for the full story and the fix.

How it works

The pattern behind most of them: we trusted a signal that looked like the truth (a tag list, a status code, a green fallback, a console change) instead of checking the real state.

Releases

What happened	Why	Do instead	Page
Two releases reverted live features, twice in one day	Production state was read from the tag list; a teammate deployed by manual run	Read ECS, run git merge-base --is-ancestor	Release safety
A tag landed on an old parked branch	A merge failed, an empty variable tagged the local checkout	Check the merge, print the commit, then tag	Release safety
A tag went out mid-way through a teammate's deploy	Only one repo's pipeline was checked	Check every production pipeline	Release safety
Whole-branch promotion would have broken staging	Branches carried drifted per-env config and missing hotfixes	Cherry-pick onto the running commit	Release safety
Screens failed during a deploy	A migration was not backward compatible	Expand and contract	Release safety
Merged code sat undeployed for hours	A merge does not deploy; the tag was missing	Push the tag, check the tag list first	Pipeline
CodeDeploy said success, traffic did not move	It swaps the listener default, not host-header rules	Check where traffic really goes	Pipeline
A deploy looked like a crash loop for 25 minutes	Boot migrations waited on table locks held by the old tasks	Run migrations as a separate task, or a longer start period	Pipeline
A hand-registered fix was reverted by the next build	The build re-registers task definitions from the repo	Commit the change, never fix by hand	Pipeline
Every build failed at install	Lockfile regenerated without a package.json change	Diff both against the last good tag	Pipeline
A web app crash-looped on staging	Health check got a 307; matcher allowed only 2xx	Real /health route, matching matcher	Pipeline

Observability

What happened	Why	Do instead	Page
Every deploy was one Sentry release	Release was a static SSM value	Per-build release baked into the task definition	Sentry
Validation errors were invisible	beforeSend dropped everything under 500; GraphQL returns 200	Log client errors with field paths, tag them	Sentry
One attacker made 600 Sentry events	Every rejected request became an event	Sample client errors	Sentry
Stack traces pointed at bundled code	Source maps never uploaded	Upload per release	Sentry
A 79% failure rate went unseen	Silent fallback plus statusCode: null in logs	Read ALB metrics, fix the serializer	Daily check
Seventeen app builds all showed one version	Only the build number moved	Bump the version name too	Crashlytics
A shipped build number differed from the archive	The upload tool renumbered it	Check which archive you sent	Crashlytics
Dev was down for hours, unnoticed	No traffic, no alarm	Health alarms in every environment	Incidents

Security and secrets

What happened	Why	Do instead	Page
User tokens and a service key were in logs	?token= and an unlisted header were logged	Redact params, headers and the URL; test it; rotate	CloudWatch and security
A token leaked through the wrong header	Redaction matched header names only	Scrub token-shaped values in every header	CloudWatch and security
No CloudTrail, no GuardDuty, no MFA	Never set up per account	Security baseline in every account	CloudWatch and security
Attackers found internal route names	API docs served in production	Turn docs off; block internal paths at the WAF	CloudWatch and security
A form's spam check failed for six hours	A secret was dropped; code fell back to another one	Fail at startup, no fallbacks	Secrets
A fixed test login code was public	It was in a repo doc and the app binary	Bypasses off in prod, rotate burned codes	Secrets
A console change would have been reverted	The parameter was Terraform-owned	Change it in Terraform	Secrets
A temporary bastion ran for a month	No end date, no owner	Temporary access expires	Incidents

AI providers and capacity

What happened	Why	Do instead	Page
Every realtime AI session died at once	Provider spend cap tripped	Check spend first; alert on quota errors	Incidents
Chat failed with auth errors in production	Per-feature keys were dead; only the base key was tested	Test every key	Secrets
A new key could not reach our model	Provider retired the model for new projects	Mint keys in the same project, or move models	Secrets
One browser opened 160,000 sockets	Backoff reset on the error frame; close code ignored	Auth close is final; refresh once	Incidents
API tasks killed with exit code 137	Unbounded sockets, each holding an AI session	Caps, load shedding, heap limit, back-pressure	Incidents

Rules

Rule

When you hit a new mistake, add it to the module page and to this table in the same change. A lesson that lives only in chat will be paid for again.

Checklist

• Read this page before your first production release.
• Read it again before your first week of production duty.
• Added any new mistake from your last incident.