Daily production check

The person on production duty runs this every working morning. It takes 20 to 30 minutes on a quiet day. It finds problems before users report them.

How it works

Go top to bottom. Each step has a "good" value. If a value is not good, check the noise list below before you raise it. Then classify it and act.

The checklist

Step	Where	Good looks like	Escalate when
Sentry	Sentry, production environment, last 24 h	No new issue with more than a handful of users. No regression.	A new issue in the latest release. Any regression.
Crashlytics	Crashlytics, each app, Android and iOS	Crash-free users flat or up versus last week.	A drop after a new version. A velocity alert.
App logs	CloudWatch Logs Insights, see queries	Errors are the known ones. Counts flat.	A new error message. A known one that doubled.
ECS	describe-services per cluster	Running equals desired. No "unhealthy" events outside a deploy.	Tasks replaced again and again. Service at max scale.
ALB	CloudWatch metrics per target group	Target 5xx near zero. p99 latency under your alarm line.	5xx above a few per hour. Latency alarm flapping.
Database	RDS metrics and Postgres error log	CPU low. Connections far below the limit.	Deadlocks, "too many connections", disk filling.
AI providers	Provider console and app logs	No quota or spend-cap errors. Spend on track.	Any RESOURCE_EXHAUSTED, spend-cap or auth error.
WAF	WAF log group	Blocks are scanners and probes.	Real users blocked. A single IP at very high volume.
GuardDuty	GuardDuty console, every account	No new finding at severity 7 or more.	Any high finding.
Pipelines	CodePipeline, every environment	Last runs Succeeded. Nothing stuck in progress.	A failed prod run. A run stuck for over 30 minutes.

Commands

Set these once in your shell. Use environment variables, not a $PROFILE string: zsh does not split P="--profile x" into two words.

export AWS_PROFILE=example-prod
export AWS_REGION=us-east-1

ECS service health and the last few events:

aws ecs describe-services --cluster example-prod --services api web \
  --query 'services[].{name:serviceName,running:runningCount,desired:desiredCount,last:events[0].message}'

ALB 5xx from the targets, per hour, for one day:

aws cloudwatch get-metric-statistics \
  --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count \
  --dimensions Name=LoadBalancer,Value=app/example-alb/0123456789abcdef \
  --start-time 2026-01-01T00:00:00Z --end-time 2026-01-02T00:00:00Z \
  --period 3600 --statistics Sum

Failed or stuck pipeline runs:

for p in $(aws codepipeline list-pipelines \
    --query 'pipelines[?starts_with(name, `example-prod-`)].name' --output text); do
  echo "$p: $(aws codepipeline list-pipeline-executions --pipeline-name "$p" \
    --max-items 1 --query 'pipelineExecutionSummaries[0].status' --output text)"
done

High GuardDuty findings:

DET=$(aws guardduty list-detectors --query 'DetectorIds[0]' --output text)
aws guardduty list-findings --detector-id "$DET" \
  --finding-criteria '{"Criterion":{"severity":{"Gte":7}}}'

Severity

Severity	Meaning	Examples
Critical	Exploitable now, or user data or credentials exposed	Tokens in logs, admin user without MFA, database port open to the internet
High	A user-facing feature is broken, or a security control is missing	Bursts of 5xx, a scheduled job failing every run, no CloudTrail
Medium	Degraded, silent data loss, or a blind spot	Quota errors, noisy logs hiding real errors, rate limiter rejecting real users
Low	Worth a ticket	A rare race condition, a cosmetic log bug

The noise list

These look scary but are normal. Do not raise them.

• ECS event "has stopped N running tasks" right after a deploy. That is blue/green draining the old tasks.
• Auto scaling "AlarmLow" alarms in ALARM state. The service is idle, so the scale-in alarm is on.
• CPU maximum near 100% at the minute the task count doubles. That is new containers booting during a deploy.
• Database "connection reset by peer" during a deploy. Old tasks are closing their connections.
• Requests for /.env, /wp-login.php, /.git/config. Internet scanners. It is only a problem if one got a 200.
• "Failed to find Server Action" in a Next.js app. An old browser tab after a deploy.
• A rate limiter returning "too many requests" at a steady low rate. The limiter is working.

Keep this list on the page and add to it. A check that cries wolf every day gets skipped.

Weekly extras

Once a week, the security owner runs the account-level checks. Logs cannot show these.

Check	Command or place	Good looks like
IAM users without MFA	aws iam generate-credential-report, then aws iam get-credential-report --query Content --output text | base64 --decode and read mfa_active	Every human has MFA, or humans use SSO only
Admin rights	IAM users and roles with AdministratorAccess	Only the people who need it
Old access keys	Same credential report, access_key_1_last_rotated	No personal key older than 90 days
CloudTrail is on	aws cloudtrail get-trail-status --name example-trail --query IsLogging	true in every account
GuardDuty is on	aws guardduty list-detectors in every region you use, and us-east-1	One detector each
Open security groups	aws ec2 describe-security-groups --filters Name=ip-permission.cidr,Values=0.0.0.0/0 --query 'SecurityGroups[].[GroupId,GroupName]'	Only the load balancer on 80 and 443
Secrets in logs	The leak queries on CloudWatch and security	Zero rows
Root account use	CloudTrail, userIdentity.type = Root	None, or confirmed by the owner
Forgotten resources	EC2 instances, bastions, idle databases	Nothing nobody can explain

Rules

Rule

The daily check is read-only. Do not restart a task, retry a pipeline, or change a WAF rule as part of the check. Write down what you found, then fix it as a separate, announced action. A "quick fix" during a scan is how scans cause outages.

Rule

Before you call a problem open, check the last time it happened, not the count. A 7-day window still counts lines from before the fix was deployed.

Rule

Never paste raw log lines into a chat or ticket. They can hold tokens and personal data. Quote counts, times with a time zone, routes, and the file and line of the cause.

Mistakes we made

• A 79% failure rate was invisible. A service rejected four of five requests, but the client fell back to a default URL and the logger wrote statusCode: null on every line. Only the load balancer metrics showed it. Always look at ALB metrics, not just app logs.
• The API logged zero 5xx while users saw errors. GraphQL returns HTTP 200 with an errors array, so "5xx from app logs" is always zero. Count GraphQL errors by code, and read 5xx from the load balancer.
• A leak looked open after it was fixed. A 7-day count showed 538 token lines, but the last one was from before the fix. Read the last timestamp.

Checklist

• All ten daily steps done, numbers written in the handover.
• Anything new is classified and either ticketed or in the incident flow.
• New noise is added to the noise list with the reason.
• On the weekly day: MFA, admin users, keys, CloudTrail, GuardDuty, open groups, leaks, root use, forgotten resources.