Incidents

An incident is anything that breaks or badly degrades production for users, or exposes data. This page is the flow to follow, and real cases from our own systems, with names removed.

How it works

Mitigate before you fix. Stopping the harm comes first. A rollback is often the fastest mitigation, because the old task definition and image are still there. See Production release safety for the command.

How to run one

Roles during an incident

Role	Job
Incident lead	Makes decisions. Usually the person on production duty.
Fixer	Changes things. Can be the lead in a tiny team.
Communicator	Posts updates. Answers "is it fixed yet" so the fixer can work.

What to post

Post when you start, at each change, and when it is over:

INCIDENT: chat replies failing for all users since 09:40 UTC.
Impact: about 100% of chat turns return an error. Other features OK.
Doing: checking AI provider key and spend. Next update 10:15 UTC.
Lead: <name>

Always say the time zone. Say what is known, what is not, and when the next update comes.

The write-up

Keep it short and blameless: describe what the system allowed, not who slipped.

1. Timeline, with time zones.
2. Impact: who, how many, how long.
3. Root cause. Not "human error": what made the mistake possible.
4. What fixed it.
5. What stops a repeat. Each item has an owner.

Case studies

AI key spend cap killed every realtime session

• Symptom. Every realtime voice session closed with an upstream error, at once. Clients reconnected in a loop. It looked like a broken feature.
• Cause. The AI provider project behind the keys had a monthly spending cap. It tripped. In another case the per-feature keys had been revoked, and only the base key worked.
• Fix. Raise the cap or swap in a working key, then restart tasks so they read it.
• Lesson. When an AI feature dies everywhere at once, check the provider spend page and key status first. Alert on quota and spend-cap errors in the logs. Test every key.

One browser made 160,000 connections in three days

• Symptom. One browser opened a WebSocket about every 1.3 seconds. On the worst day it was 89% of all realtime traffic, every connection refused for a bad token.
• Cause. Three client bugs together. The error frame the server sent before closing reset the reconnect backoff to zero. The client ignored the close code that meant "token invalid". And signed-in users were reconnected with the same expired token.
• Fix. Server: after 5 refusals of the same token in 10 minutes, refuse the upgrade with a plain 403 and no frame, so the client's backoff and attempt cap work. Client: treat an auth close as final, refresh the token once, then reconnect once; add jitter to backoff.
• Lesson. Reset backoff only on proof of a healthy session, never on any message. Give auth failures their own close code, and make clients respect it.

Exit code 137: memory crash loop

• Symptom. API tasks were killed and replaced over and over. ECS showed exit code 137, which means the container was killed for using too much memory.
• Cause. A reconnect stampede on realtime routes. Every socket held an upstream AI session. There was no cap on sockets, no back-pressure on send buffers, and a 100 MiB default max message size. Memory grew until the container limit killed the task.
• Fix. Cap concurrent sessions per task and per user, and refuse with 503 and Retry-After before the socket opens. Shed load with 503 when event-loop delay, heap or memory is high, but keep /health answering. Set Node's heap limit below the container limit (1536 MB on a 2048 MB task). Cut max message size to 256 KiB. Close sockets whose send buffer grows too large, and drop silent peers with a heartbeat.
• Lesson. Every long-lived connection needs a limit, a timeout and back-pressure from day one. The default limits of a library are not safe limits.

A region-lookup service rejected 79% of requests, and nobody saw

• Symptom. None from users. Load balancer metrics showed 13,600 of 17,300 requests in a day were 4xx.
• Cause. The caller was a web app's server-side render, not browsers. It said it was a browser client, and that client type needs a browser challenge token that a server can never produce, so every call got 401. A failed lookup was dropped from the cache, so every page render tried again. The caller silently fell back to a default URL, so users saw nothing. And the logger wrote statusCode: null on every line, so logs could not show the failure rate.
• Fix. Fix the logger serializer, so the failure is measurable. Give server callers their own way to authenticate instead of pretending to be a browser.
• Lesson. A silent fallback hides a broken path. Measure from the load balancer as well as from app logs. Sample the failing requests before you guess who sends them: the first guess here was wrong.

A release built on an old base reverted live work

• Symptom. A feature disappeared from production right after an unrelated release, came back, then disappeared again after the next one.
• Cause. The deployed state was read from the tag list. A teammate had deployed by manual pipeline run, which leaves no tag. Two releases were built from a base older than production.
• Fix. Roll back to the last good task definition. Rebuild the release on top of what was running.
• Lesson. Read ECS, and check git merge-base --is-ancestor before every production release. Full story on Production release safety.

A retired AI model took down chat in dev, unnoticed

• Symptom. In dev, chat was fully down. Nobody noticed: there was no dev chat traffic for hours. In production, a hand-set model override broke the fast path and safety gate, but the main reply still worked.
• Cause. The provider stopped serving the configured model to new projects (404). A newer model rejected a hardcoded setting. Model names were set both in the repo config and in task definitions registered by hand.
• Fix. Move every env to current models, through the repo config, then remove the hand overrides.
• Lesson. Keep model names in one place in the repo. Check what each env actually runs. An unused dev env can be broken for days.

Rules

Rule

Mitigate first, then find the root cause. If the problem started with a deploy, roll back without debating it. You can always roll forward again.

Rule

One incident lead at a time, named in the channel. Two people changing production at once during an incident makes it worse and destroys the timeline.

Rule

Every Critical or High incident gets a written, blameless write-up, and every lesson lands in a runbook or this handbook. Lessons that only live in chat get lost.

Mistakes we made

• We debugged code for an outage caused by a spend cap. Check the provider first.
• Dev was fully down for hours and nobody noticed. No traffic, no alarm. An environment nobody watches can be broken for days. Point at least the health alarms at every environment.
• A slow deploy looked like a crash loop. New tasks were replaced again and again for 25 minutes. The old tasks served users the whole time, so there was no outage. Before you declare an incident on a deploy, check whether users are affected.
• A forgotten bastion ran for a month. A temporary EC2 host for database access was never removed, and its role could read secrets. Temporary access gets an end date and an owner.

Checklist

• Incident lead named in the channel.
• First update posted: impact, start time with zone, next update time.
• Harm stopped: rollback, flag off, block, scale, or key swap.
• Metrics and users confirmed back to normal.
• Root cause fixed, or ticketed with an owner.
• Write-up done within 2 working days.
• Runbook and handbook updated.