Cost and clean-up
What it is
A monthly look at what each environment costs and why, choosing the right size for each resource, and a safe way to retire things: archive, verify, then destroy.
Why we do it
We found whole stacks running for weeks with no DNS pointing at them, paid-for network endpoints nothing used, load-test database clones still running after the test, and a staging environment costing 30% of the whole AWS bill. Every one was found by reading the bill.
Steps
1. The monthly cost review
- Cost Explorer, group by account, then by service, last full month.
- For each line bigger than about 5% of the total, say what it is and why it is that big.
- Estimate the run rate from the last 8 days' daily average times 30, not the calendar month, which hides recent changes.
- Compare with last month. Any line up more than 20% gets an explanation.
- Record the totals and decisions in the project's cost log.
2. Rightsizing rules we use
| Choice | Our rule | Why |
|---|---|---|
| Fargate Spot | Dev only, never prod | A Spot reclaim in prod is a user-visible outage |
| Dev and sandbox boxes | Stop when unused, do not terminate | A stopped box costs only its disk |
| Staging database | The smallest size that still rehearses prod; consider serverless | Usually the largest saving available |
| NAT gateways | One per AZ in prod; one total is acceptable in dev | One NAT is a single point of failure |
| VPC endpoints | Only for services the private tasks really call | Unused endpoints cost about as much as a small server |
| Memory floor | Size from real peaks, not averages | Our API ran at about 350 MB normally and was killed at 1 GB in a spike |
3. Retiring something safely
- Prove it is unused: no DNS record, near-zero requests in the access logs for two weeks, no other service calling it.
- Archive to the backup bucket: database dump, uploaded files, secret values (encrypted), Terraform state.
- Verify the archive: table and row counts match, one restore test.
- Destroy with Terraform. Expected blockers: database deletion protection, non-empty image repositories, non-empty buckets.
- Sweep leftovers that Terraform does not own: parameters created by hand, log groups, DNS records at the registrar, IAM users or groups, certificates, third-party workspaces.
Check it worked
The next month's bill shows the line gone. The inventory row is marked retired with the archive location.
Rules
Archive and verify before you destroy. Once we found the archive was the only surviving copy of some data.
Temporary resources get a DeleteAfter tag when created. Load-test clones, demo boxes and
bastions are the usual forgotten ones.
Mistakes we made
- A whole stack ran for weeks with no DNS pointing at it. Its biggest cost was network endpoints. Retired after archiving.
- A dev stack served only scanner traffic for weeks before being destroyed.
- Load-test database clones kept running and costing money after the test.
- Paid VPC endpoints for a service nothing called.
- Retiring a stack left behind hand-made parameters, an IAM group with a person still an admin, and a DNS record at the registrar.
Done when
- Monthly cost review done and recorded.
- Dev boxes stopped when idle; Spot only in dev.
- Retired resources archived, verified, destroyed and swept.