CloudWatch and account security

CloudWatch holds our application logs, metrics and alarms. CloudTrail and GuardDuty watch the AWS accounts themselves. This page shows how they fit, the queries we use, and the security baseline every account needs.

How it works

CloudWatch Logs stores log lines in log groups, one per service per environment. Logs Insights is the query language for searching them. Alarms watch a metric and notify an SNS topic (a message channel) when it crosses a line.

CloudTrail records every AWS API call: who changed what, and when. GuardDuty reads CloudTrail, DNS and network logs and raises findings when something looks like an attack.

How to build it

Log groups

• One log group per service per environment: /app/<env>/<service>.
• Set retention on purpose. We keep app logs 14 days. Longer costs more; shorter loses evidence. Decide, and write down why.
• Log JSON, one object per line, with a numeric level. With pino, 40 is warning, 50 is error, 60 is fatal. Queries can then filter level >= 50.

Logs Insights queries

Run these from the CloudWatch console, or with aws logs start-query. Logs Insights charges per GB scanned, so pick the shortest time window that answers the question.

Errors per service per day:

filter level >= 50
| stats count(*) as n by @log, level, datefloor(@timestamp, 1d) as day
| sort @log, day

Top errors with the database cause pulled out of the stack:

filter level >= 50
| parse err.stack /caused by: (?<cause>.{0,160})/
| stats count(*) as n, min(@timestamp) as first, max(@timestamp) as last
    by @log, msg, coalesce(cause, substr(err.message, 0, 160)) as reason
| sort n desc

Leak check: tokens in request URLs that were not redacted:

filter msg = 'incoming request' and req.url like /token=/
  and req.url not like /token=\[REDACTED\]/
| parse req.url /^(?<route>[^?]+)/
| stats count(*) as n, max(@timestamp) as last by @log, route

Leak check: other secret-shaped strings anywhere in a line:

filter @message like /(Bearer ey|"password":"|AKIA[0-9A-Z]{16}|api[_-]?key=)/
| stats count(*) as n by @log, msg
| sort n desc

Postgres errors, without known migration noise:

filter @message like /(FATAL|ERROR|PANIC)/ and @message not like /already exists/
| parse @message /:(?<sev>FATAL|ERROR|PANIC):\s+(?<m>.{0,140})/
| stats count(*) as n, max(@timestamp) as last by @log, sev, m
| sort n desc

Read the last column first. A count includes lines from before a fix went live.

Alarms

A minimum set per environment, all to one SNS topic:

Alarm	Metric	Why
Target 5xx rate	HTTPCode_Target_5XX_Count divided by RequestCount	The app is failing requests
Load balancer 5xx	HTTPCode_ELB_5XX_Count	No healthy target, or a timeout
p99 latency	TargetResponseTime, p99	Slow requests users feel
Unhealthy hosts	UnHealthyHostCount above 0	Tasks failing health checks
At max scale	ECS RunningTaskCount at the autoscaling maximum	You have run out of room
Database CPU and connections	RDS metrics	Saturation before an outage
Cache memory	ElastiCache memory and evictions	Cache about to drop keys

Email subscriptions to SNS must be confirmed from the opt-in email, or they get nothing.

Account security baseline

Every AWS account, including dev and sandbox accounts:

1. CloudTrail: one multi-region trail, log file validation on, logs to an S3 bucket with 365-day retention.
2. GuardDuty: a detector in every region you use, plus us-east-1. IAM is a global service and its events are recorded in us-east-1, so a check that only reads your main region misses CreateUser and CreateAccessKey.
3. Findings to people: an EventBridge rule for GuardDuty severity 7 or more from every account, sent to one Lambda that posts to the security channel. Test it with a sample finding.
4. MFA: every human has MFA. Better: humans sign in through SSO only, and there are no human IAM users.
5. Least access in production: developers get view-only plus log read in production, not ReadOnlyAccess. ReadOnlyAccess can read S3 objects, which may hold user data, and can read SSM SecureString secrets encrypted with the default AWS key.

Moving humans to SSO

SSO (IAM Identity Center, with your company identity provider) means people sign in with their work account and its MFA, and get short-lived credentials.

• Phase A adds a login path and breaks nothing. Phases A to C can be undone. Phase D cannot.
• Keep the same CLI profile names. Only their definition in ~/.aws/config changes to sso_session, so scripts and Terraform backends need no edits.
• Remove a person's password only after they have signed in through SSO. Doing it earlier locks them out.
• Service users and workload roles (ECS, CodeBuild, Lambda) are not part of this. Move them in a separate pass.
• Confirm the identity provider actually enforces MFA. If it does not, SSO is single factor.

Rules

Rule

Never log secrets or tokens. Redact by header name and by query parameter, and also scrub the full URL string. Then test it: send a fake token and a fake key, and assert neither appears in the log output. Anyone with log read access can replay a logged token.

Rule

When a secret was logged, rotate it after the redaction fix is live. Redaction stops new leaks. It does not make the old lines safe. User session tokens expire on their own; service keys do not.

Rule

CloudTrail and GuardDuty are on in every account, from the day the account is created. Without them you cannot answer "who did this" after an incident.

Mistakes we made

• User tokens and a service key were in the logs for weeks. Realtime routes (SSE and WebSocket) take the token in ?token= because browsers cannot set an auth header there. The logger redacted authorization but logged the full URL and query. An internal service key header was not on the list at all. Fix: add the query param and the header to the redact list, scrub the URL string, then rotate the key.
• A fixed redact list missed a token in the wrong header. A malformed curl put Bearer ... inside Content-Type. The list matched header names, so it never fired. Fix: also scrub any JWT-shaped or Bearer value from every header.
• The logger wrote statusCode: null on every line. A pino serializer expected a field the Fastify reply object does not have. 4xx and 5xx rates were unmeasurable from logs. Use the framework's own reply serializer.
• No CloudTrail, GuardDuty off, every human an admin with no MFA. Found on the first full scan. Turning on the trail and detectors took one Terraform module. The human access fix needs the SSO rollout above.
• Outside attackers found internal routes from the API docs. A service on the public load balancer served its OpenAPI document, which listed the internal route names. Nothing got in, but they knew where to aim. Turn off API docs in production, and block internal paths at the WAF unless the source is your own NAT address.

Checklist

• Each service has its own log group with a chosen retention.
• Logs are JSON with a numeric level.
• Redaction covers auth headers, service key headers, ?token=, and the URL string, with a test.
• The leak queries return zero rows.
• The alarm set exists and the SNS email subscriptions are confirmed.
• CloudTrail multi-region trail logging, in every account.
• GuardDuty in every used region and us-east-1, high findings reach the security channel.
• Every human has MFA, or signs in through SSO only.
• Developers have view-only plus logs in production.