Skip to main content

Production release safety

The most important page in this module. A production release can silently remove work that is already live. This page shows how that happens and the checks that stop it. Run every check, every release, even when you are in a hurry.

How it works​

This is what happened to us, with names removed.

Two things went wrong. First, the tag list was taken as the truth about production. It was not: manual pipeline runs leave no tag. Second, a release was built on a base older than what was running. Any release like that reverts everything in between, no matter how the earlier deploy was triggered.

Features X and Y were never broken. Their code was simply missing from the image serving traffic. The database migrations it needed stayed applied, which was harmless: the schema was just ahead of the code. A plain image rollback fixed it completely.

The checks​

1. Read what production runs​

Never infer it from tags, branch names or chat. Ask ECS. For a service deployed by CodeDeploy, the live task definition is on the PRIMARY task set.

export AWS_PROFILE=example-prod AWS_REGION=us-east-1

TD=$(aws ecs describe-services --cluster example-prod --services api \
  --query 'services[0].taskSets[?status==`PRIMARY`].taskDefinition' --output text)
aws ecs describe-task-definition --task-definition "$TD" \
  --query 'taskDefinition.containerDefinitions[0].image' --output text
# -> 123456789012.dkr.ecr.us-east-1.amazonaws.com/example-app/api:abc1234

The image tag is the first 7 characters of the commit. Do this for every service built from the same repo, for example a public API and an admin API. They can run different images.

2. Confirm the release contains what is live​

RUNNING=abc1234                  # from step 1
RELEASE=$(git rev-parse HEAD)    # the commit you are about to tag
git fetch origin
git merge-base --is-ancestor "$RUNNING" "$RELEASE" \
  && echo "safe: release contains what is live" \
  || echo "STOP: this release REVERTS production"

git merge-base --is-ancestor X Y succeeds only if commit X is in the history of Y. If the running commit is not in your release's history, your release does not contain it.

3. Check every production pipeline​

Check all of them, not only the repo you are tagging. Teammates deploy in parallel.

for p in $(aws codepipeline list-pipelines \
    --query 'pipelines[?starts_with(name, `example-prod-`)].name' --output text); do
  s=$(aws codepipeline list-pipeline-executions --pipeline-name "$p" --max-items 1 \
    --query 'pipelineExecutionSummaries[0].status' --output text)
  [ "$s" = "InProgress" ] && echo "IN PROGRESS: $p"
done

4. Fetch again right before you tag​

Production can change in the hour between your first check and your tag. Re-run step 1 and step 2 immediately before pushing the tag, not once at the start of the release.

5. Guard the tag against a failed merge​

A tag deploys whatever commit it points at. If a merge fails and your script keeps going, an empty variable makes git tag point at your local checkout, which may be an old branch.

C=$(gh pr view <pr-number> --json state,mergeCommit \
  --jq 'select(.state=="MERGED") | .mergeCommit.oid')
[ -n "$C" ] || { echo "STOP: PR is not merged"; exit 1; }

git fetch origin
git merge-base --is-ancestor "$C" origin/main || { echo "STOP: not on main"; exit 1; }

git log -1 --oneline "$C"        # read it. Is this the commit you mean?
git tag v-1.4.0 "$C"
git push origin v-1.4.0

Never chain merge and tag in one unguarded command.

Promote by cherry-pick onto what is running​

When branches have drifted, merging a whole branch into main can carry config or revert hotfixes you did not mean to touch. Build the release on top of the running commit and add only the commits you are shipping:

git checkout -b promote/feature-x abc1234        # the commit production runs
git cherry-pick <squash-commit-1> <squash-commit-2>
git diff abc1234 HEAD --name-only                 # only files in this release?

If the diff shows files outside the release, stop and find out why.

Roll back without a rebuild​

The previous task definition revision and its image are still there. Point CodeDeploy at the last good revision:

aws deploy create-deployment \
  --application-name example-prod-api \
  --deployment-group-name example-prod-api-dg \
  --revision '{"revisionType":"AppSpecContent","appSpecContent":{"content":"{\"version\":0.0,\"Resources\":[{\"TargetService\":{\"Type\":\"AWS::ECS::Service\",\"Properties\":{\"TaskDefinition\":\"arn:aws:ecs:us-east-1:123456789012:task-definition/example-prod-api:41\",\"LoadBalancerInfo\":{\"ContainerName\":\"api\",\"ContainerPort\":3000}}}}]}"}}'

This takes minutes. Find the last good revision with aws ecs list-task-definitions --family-prefix example-prod-api --sort DESC, and confirm its image with step 1 before you deploy it. After the rollback succeeds, read the running image again.

Rules​

Rule

Before any production release, read what ECS is running and confirm the release commit contains it with git merge-base --is-ancestor. The newest tag is not what production runs. A release built from an older base reverts everything in between.

Rule

Run the ECS check again immediately before you push the tag. Production moved under us in the hour between a check and a tag.

Rule

Only tag a commit you have printed and read, and that is on the target branch. A failed merge plus an empty variable tags whatever your checkout points at.

Rule

Migrations must work with both the old and the new code. Add new columns first, switch the code, remove old columns in a later release. This is called expand and contract. During blue/green the old tasks keep serving while the new schema is applied.

Rule

If a fix has a data part and a code part, and someone says "push it to prod", confirm which part they mean before you deploy code. A production code deploy is its own decision.

Mistakes we made​

  • Two releases in one day reverted live work, twice. Described above. The fix is the two checks in steps 1 and 2. Asking everyone to deploy by tag helps too, but it is not enough on its own.
  • We told the team features were "undeployed" when they were live. The decision about what to ship was then made on false information. State what is deployed only after running step 1.
  • A tag went on a parked branch. A merge failed because of a conflict, the script went on, and git tag with an empty commit tagged an old local checkout. The pipeline started building it. It was caught at the Source stage and the tag deleted.
  • A tag went out while a teammate's release was mid-deploy. It was only one repo's pipeline being checked. Check all production pipelines.
  • Whole-branch promotion would have broken staging and reverted prod. One branch had a drifted staging task definition that turned a feature off. Another lacked a hotfix migration that production had. Promote by cherry-pick onto the running commit.
  • A schema change broke screens during a deploy. A migration that was not backward compatible ran while old tasks still served traffic. 11 requests failed with "column does not exist". Use expand and contract.

Checklist​

  • Running image read from ECS for every service built from this repo.
  • git merge-base --is-ancestor <running> <release> passed.
  • No production pipeline in progress, in any repo.
  • main fetched again, ECS read again, right before the tag.
  • The tagged commit is merged, on the branch, and printed.
  • Migrations in this release are expand-and-contract safe.
  • Last good task definition revision noted for rollback.
  • After deploy: pipeline Succeeded, running image equals the release commit.
Last updated on