Skip to main content

Secrets and config

Every secret our services use lives in AWS SSM Parameter Store. This page shows the layout, how a secret reaches a running container, how to rotate one, and the traps we fell into.

How it works​

SSM Parameter Store is an AWS key-value store. A SecureString parameter is encrypted with KMS. The task definition does not hold the secret. It holds a pointer (valueFrom), and ECS fetches the value when it starts the task.

How to build it​

Layout​

One path per environment and service:

/app/<env>/<service>/<KEY>

/app/dev/api/DATABASE_URL
/app/dev/api/JWT_SECRET
/app/prod/api/DATABASE_URL
/app/prod/api/AI_PROVIDER_API_KEY

The execution role of each service can read only its own prefix. Then a leak in one service does not expose another service's secrets.

What goes where​

KindWhereExample
SecretSSM SecureString, task definition secretsDatabase URL, JWT key, API keys
Per-env plain valueTask definition environmentFeature flags, region, bucket name, the Sentry release
Non-secret app configA config file in the repoModel names, timeouts, rate limits

The task definition​

"secrets": [
  {
    "name": "DATABASE_URL",
    "valueFrom": "arn:aws:ssm:us-east-1:123456789012:parameter/app/prod/api/DATABASE_URL"
  }
],
"environment": [
  { "name": "AWS_REGION", "value": "us-east-1" }
]

Placeholders filled by hand​

Terraform creates secrets it cannot generate (a third-party key, for example) as a placeholder, and ignores later changes to the value:

resource "aws_ssm_parameter" "provider_key" {
  name  = "/app/prod/api/AI_PROVIDER_API_KEY"
  type  = "SecureString"
  value = "PLACEHOLDER"

  lifecycle {
    ignore_changes = [value]
  }
}

Then a human sets the real value. Terraform never overwrites it:

aws ssm put-parameter --name /app/prod/api/AI_PROVIDER_API_KEY \
  --value "..." --type SecureString --overwrite

A parameter without ignore_changes is owned by Terraform. If you change it in the console, the next terraform apply puts the old value back. Change it in Terraform instead.

Rotating a secret​

Tasks read SSM only at start. Changing the value does nothing to running tasks. For a service deployed by CodeDeploy blue/green, update-service can change only a few fields, and forcing a new deployment is not one of them. Stop the tasks one at a time instead: ECS replaces each one, and the new task reads the new value. With three or more tasks there is no downtime.

aws ecs list-tasks --cluster example-prod --service-name api --query 'taskArns' --output text
aws ecs stop-task --cluster example-prod --task <task-arn> --reason "secret rotation"
# wait until the replacement is healthy, then the next one

If two services share a secret (an API and its admin twin, or a caller and a callee), rotate and restart both together.

Checking a value without printing it​

Never print a secret to compare it. Compare a short hash:

aws ssm get-parameter --name /app/prod/api/JWT_SECRET --with-decryption \
  --query Parameter.Value --output text | shasum -a 256 | cut -c1-10

Rules​

Rule

Secrets never go in taskdef.json values, in git, in a Dockerfile, or in a chat. Always reference SSM through secrets[].valueFrom. Anything in git is readable by everyone with repo access, forever, even after you delete it.

Rule

A missing required secret must stop the app at startup with a clear error. Never fall back from one secret to another with ?? or ||. A silent fallback turns a config mistake into a hidden outage.

Rule

Bypass and test switches (skip attestation, fixed review login, test OTPs) are off in production, and the code refuses them there even if someone sets them. A test code committed to git or shipped in an app binary is burned. Rotate it.

Rule

When an AI feature dies everywhere at once, check the provider's spend page and key status before you debug code. A spending cap or a revoked key looks exactly like a broken feature.

Mistakes we made​

  • A spending cap looked like a code bug. AI provider keys sat on a project with a monthly spend cap. When it tripped, every realtime session closed with an upstream error and clients reconnected over and over. Check spend first, and put an alert on quota and spend-cap errors in the logs.
  • Only one of three AI keys was ever tested. The service used a base key plus one key per feature, to split quota. Only the base key was checked at setup. The per-feature keys were dead, and chat failed in production with an auth error. Test every key you put in.
  • A new key could not use the model we needed. The provider had stopped offering an older model to new projects. A key from a fresh project returned 404 on that model. Mint new keys inside the same provider project, or move to a current model first.
  • A secret disappeared and another one took its place. A task definition revision dropped one secret from secrets. The code had SECRET_A ?? SECRET_B, so it silently used a different widget's secret. A public form's spam check failed for about six hours. No error at startup, no alarm.
  • A console change was reverted by Terraform. A flag was changed by hand in SSM. That parameter had no ignore_changes, so the next apply would have put the old value back. Change Terraform-owned values in Terraform.
  • A fixed test code was public. A store-review login used a fixed one-time code that was in a repo doc, on an account whose email was in the app binary. Anyone could sign in as that account. It was rotated, then turned off when reviews were done.

Checklist​

  • Every secret is a SecureString at /app/<env>/<service>/<KEY>.
  • Task definitions reference secrets by valueFrom. No secret values in git.
  • Each execution role reads only its own prefix.
  • Hand-filled placeholders have ignore_changes = [value]. Terraform-owned values are changed in Terraform.
  • The app fails at startup when a required secret is missing. No fallbacks between secrets.
  • Every new key is tested on its own before it goes in.
  • After a rotation: tasks restarted, no auth errors, old value revoked.
  • AI provider spend caps known, and quota errors alert someone.
Last updated on