Monitoring and alerts
What it is
CloudWatch collects metrics and logs from AWS services. Alarms watch a metric and
fire when it crosses a line, for example "more than 5% of API responses are errors for 5
minutes". Alarms send to an SNS topic, which forwards them to the team chat and email.
An uptime check calls /health from outside every minute.
Saved log queries and the security baseline are in CloudWatch and account security.
Why we do it
Without alarms, users are the monitoring. Every outage we have had was visible in a metric minutes before anyone noticed.
Before you start
- The service is deployed to dev (step 6).
- The team chat channel for alerts.
How it fits together
Steps
1. The alert topic
Create an SNS topic per environment (acme-prod-alerts), subscribe the team mailbox, and
connect the team channel with Amazon Q Developer in chat applications (formerly AWS
Chatbot), which supports Slack and Microsoft Teams.
2. The starter alarms
Create these in Terraform for every service, with the prod thresholds below and looser ones in dev:
| Alarm | Metric | Fires when |
|---|---|---|
| API errors | Load balancer HTTPCode_Target_5XX_Count over RequestCount | Above 5% for 5 minutes |
| Unhealthy tasks | Target group UnHealthyHostCount | Above 0 for 3 minutes |
| Slow API | Target group TargetResponseTime p95 | Above 2 seconds for 10 minutes |
| Service memory | ECS MemoryUtilization | Above 85% for 10 minutes |
| Database CPU | RDS CPUUtilization | Above 80% for 15 minutes |
| Database storage | RDS FreeStorageSpace | Below 20% |
| Database connections | RDS DatabaseConnections | Above 80% of the maximum |
| Budget | AWS Budgets | 80% and 100% of the month's budget |
3. The uptime check
A Route 53 health check, or a CloudWatch Synthetics canary, calling
https://api.example.com/health every minute, with an alarm on failure. It catches what
internal metrics miss: DNS, certificates, the CDN.
4. A runbook line per alarm
Each alarm's description says what to check first, for example "Check the latest deploy in CodeDeploy; if it started in the last 30 minutes, roll back." The person woken at 3 a.m. should not have to think.
Check it worked
Use Set alarm state to force each alarm into ALARM in dev, and confirm the message
reaches the channel and the mailbox:
aws cloudwatch set-alarm-state --profile acme-dev \
--alarm-name acme-dev-api-5xx --state-value ALARM --state-reason "test"
If it goes wrong
| You see | It means |
|---|---|
| Alarm fires but nobody hears | The SNS subscription was never confirmed; check the mailbox |
| Alarms fire all night in dev | Thresholds copied from prod; loosen dev or mute it out of hours |
Alarm stuck in INSUFFICIENT_DATA | Wrong metric dimensions, for example the target group name |
Done when
- Alert topic per environment, connected to chat and email.
- Starter alarms on every service and the database.
- Uptime check on
/health. - Each alarm has a runbook line; each was tested.