Mistakes

Every architecture mistake from this module in one place. Each one cost real time, or would have caused an outage if we had not caught it. Read this list before a design review.

How it works

Most of these mistakes share one of four root causes.

The fix for each root cause:

Root cause	Fix
Tested with one region only	Write a synthetic "foreign country" test now. Check every module for multi-region support while nothing depends on it.
Live setup and code drifted apart	No console changes. Read the full plan. Build databases from empty in CI.
Pre-prod did not match prod	Staging has prod's shape, prod's network path and prod's security checks.
A label nobody re-checked	Before you rely on a description ("content only", "VPC cost", "WAF on"), go and look.

How to build it

Use this list in design reviews. For each item, ask: could my change repeat it?

Data and residency

• The "global" tier held user content and staff accounts. Group chat, posts, membership lists, moderation reports and staff accounts all lived in the tier we planned to copy to every region. Why: one group's timeline needs every member's posts in one place. Instead: list every schema in the global tier and treat replication as a privacy decision. Data residency
• User files shared the global content bucket. Instead: a regional bucket for user files from day one. Data residency
• Home region could never change. Instead: build an operator-run region transfer before region two. Data residency
• Guest answers were saved before the region was known. Instead: keep them on the device until the region is confirmed. Data residency
• A fresh content database could not be built. Tables were renamed by hand in every environment. Instead: every change is a migration, and CI builds from empty. Service shape
• New migrations were silently skipped. A release shipped without a table and staff sign-in failed. Instead: the runner applies every new file and fails loudly. Service shape

Discovery and sign-in

• Header name mismatch caused 401s on every request. The app's fallback hid it on start-up, but sign-in had no fallback. Instead: curl with the exact headers the client sends. Region discovery
• Our own server-side rendering caused a 401 storm. It called as a "web" client, which needs a browser token, and retried every render. Instead: a server client type with a shared secret, and backoff on failure. Region discovery
• Logs showed statusCode: null. The failure rate was only visible in load balancer metrics. Instead: check that logs show real status codes. Region discovery
• Attestation only ran in prod. Instead: staging runs it for real. Region discovery
• No country to region mapping. Unknown countries silently fell back to the default region. Instead: list countries per region, log unmapped ones. Region discovery
• Mobile fell back to a hardcoded region. Instead: fail closed like the web client. Region discovery
• Discovery, its registry and its map ran in one region. That region down means sign-in down everywhere. Instead: per-region discovery behind latency routing, a global table, one source for every map. Region discovery
• Test-build routing was lost at sign-in. Instead: apply routing rules in both discovery routes. Region discovery

Identity and product surfaces

• Forked web repos almost deployed over a live app. They carried the original's container repository, names and config prefix. Instead: rename five values before the first tag. One identity, many apps
• An old hostname stayed attached to the wrong app. Instead: remove old rules and records when a hostname changes owner. One identity, many apps
• A dev flag widened the staff permission gate. Instead: the audience check is unconditional. One identity, many apps

Accounts, DNS and Terraform

• A plain apply would have broken prod email sign-in. Live fixes were never put in code. Instead: -target only until reconciled, and no console changes. AWS accounts and environments
• Staging's network was built by hand. Instead: every environment from the same modules. AWS accounts and environments
• Importing a policy deleted a permission. Instead: diff live against the module before importing. AWS accounts and environments
• One hand-made parameter failed a whole apply. Instead: import anything that already exists. AWS accounts and environments
• Importing a task definition deregistered the running one. Instead: never import task definitions. AWS accounts and environments
• A zone rebuilt from guesses missed five records. Instead: start from the registrar's zone export. AWS accounts and environments
• The apex could not point at a load balancer. Instead: move the apex zone to Route 53 and keep the registration. AWS accounts and environments
• A dig +short check compared nothing. Instead: compare full output. AWS accounts and environments
• Admin users had no MFA, and there was no CloudTrail. Instead: security baseline on day one, then Identity Center. AWS accounts and environments
• A new account's Fargate quota blocked scaling. Instead: raise quotas per account and region before launch. Scaling to a new region

Network and edge

• mTLS on the shared listener would have locked out mobile. Instead: a separate load balancer for browser apps. Network and edge
• The WebSocket idle timeout lived only on a branch. Instead: merge infra with the feature. Network and edge
• We blamed NAT for an endpoint bill. Instead: check the usage type first. Network and edge
• We paid for an endpoint nobody used. Instead: check a private service calls it. Network and edge
• Plain HTTP timed out everywhere. Port 80 was closed in the security group. Instead: open it where a redirect listener exists. Network and edge
• WAF only counted. Instead: Count has an end date, then Block. Network and edge
• The cheapest CDN price class had no edge near users. Instead: pick by user location. Network and edge
• Blue/green swapped the default action, not our host rules. Instead: make the deploy swap the rule traffic uses. Network and edge

Service shape

• A duplicate GraphQL type name crash-looped the API. Thousands of tests were green. Instead: one test that builds the full merged schema. Service shape
• Two services from one image drifted apart. Instead: one source for both task definitions, and verify on the service that serves the screen. Service shape
• Rollback did not restore the schema. Instead: additive migrations only. Service shape
• A placeholder image restart-looped and cost money. Instead: desired count 0 until the first real image. Service shape

Rules

Rule

When you hit a new mistake that cost time or caused an outage, add it here and to the page it belongs to, the same week. Lessons that live only in chat get lost.

Mistakes we made

This whole page is the list. Add new ones above.

Checklist

• I read this list before my design review.
• My change does not repeat any item on it.
• If I found a new mistake, I added it here and on its page.