Release pipeline

This is how a code change gets from a branch to running containers. It uses AWS CodePipeline, CodeBuild and CodeDeploy, and it starts only when someone pushes a git tag. Read it before you set up a new service or debug a failed deploy.

How it works

Blue/green means the old version (blue) keeps running while the new one (green) starts next to it. Traffic moves only when green is healthy. If green fails, blue never stopped, so users see nothing.

A tag is the only trigger. A merge deploys nothing. This is on purpose: the tag says exactly which commit an environment runs.

How to build it

Branches and tags

Branch	Environment	Tag pattern	Example
dev	Dev	dev-v-*	dev-v-0.12.0
staging	Staging	staging-v-*	staging-v-0.12.0
main	Production	v-*	v-1.4.0

One feature branch goes to dev, then staging, then main, by pull request each time. All merges are squash merges. Production hotfixes branch from main, merge to main, and are then cherry-picked to staging and dev.

git tag dev-v-0.12.0 <merge-commit>
git push origin dev-v-0.12.0

Starting the pipeline by hand from the console also works, but it leaves no version marker. Nobody can tell afterwards what the environment runs. Prefer the tag.

The four files every service needs

File	What it is
buildspec-ci.yml	Build steps for PR checks
buildspec.yml	Build steps for the pipeline: test, build image, push, write deploy files
appspec.yml	Tells CodeDeploy which container and port take traffic
taskdef.json	The ECS task definition template

# appspec.yml
version: 0.0
Resources:
  - TargetService:
      Type: AWS::ECS::Service
      Properties:
        TaskDefinition: <TASK_DEFINITION>
        LoadBalancerInfo:
          ContainerName: api
          ContainerPort: 3000

Keep one task definition per environment in the repo, for example deploy/dev/taskdef.json and deploy/prod/taskdef.json. The build picks the right one and copies it to taskdef.json. Then a merge from dev to staging never changes another environment's settings.

Speed knobs

A deploy has two slow parts: the build, and waiting. These settings change both.

Knob	Dev	Staging	Prod	Effect
CodeBuild compute size	Medium	Medium	Medium	Tests ran about 2.5 times faster than on Small, for the same cost per build
Blue termination wait	0 min	0 min	5 min	Minutes blue stays alive after traffic moves. Added to every deploy
Green health-check interval	30 s	30 s	30 s	Changed nothing for us; lowering it in dev caused recurring 502s

Bumping the Fargate task size does not make deploys faster. The levers are build size and wait time.

Why prod keeps the 5-minute wait. For those 5 minutes, rollback is one click: traffic goes straight back to blue. After blue is gone, rollback means a new deploy of the old task definition, which takes a few minutes. We have no alarms attached to the deploy groups yet, so this window is production's only fast safety net. Remove it only after alarm-based automatic rollback exists.

Rules

Rule

Use cut -c1-7 for substrings in buildspec.yml, never ${VAR:0:7}. CodeBuild's standard image runs commands with dash, not bash, and dash fails with "Bad substitution".

Rule

In taskdef.json, CodeDeploy replaces only <IMAGE1_NAME>. Hardcode everything else: the execution role ARN, the task role ARN, the region, the log group. A literal <EXECUTION_ROLE_ARN> passes straight through and the deploy fails with "unable to assume role". Any other placeholder, such as <SENTRY_RELEASE>, must be replaced by your own sed in the build.

Rule

The task execution role needs explicit ssm:GetParameters and KMS decrypt permissions for the secrets in the task definition. The AWS managed execution policy does not include them, and tasks fail to start without them.

Rule

Do not set the ECR repository to IMMUTABLE if the build uses a BuildKit registry cache. BuildKit pushes the same buildcache tag on every build, and immutable tags reject the overwrite. Use MUTABLE, or immutable with an exclusion for the cache tag.

Rule

Every task definition lives in the repo. Never register one by hand as a fix. The next build registers a new revision from the repo file and silently undoes your change.

Mistakes we made

• Merged code sat undeployed for hours. People thought the webhooks were broken and started pipelines by hand. They were never broken: a merge does not deploy, and the tag was missing. Check the tag list before you suspect infrastructure.
• CodeDeploy reported success but traffic did not move. Blue/green swaps the listener's default action. Our routing used host-header listener rules, which CodeDeploy did not touch, so traffic still went to the empty target group. Know which listener or rule really carries your traffic, make sure Terraform does not fight CodeDeploy over it, and check where traffic goes after the first deploy.
• A new pipeline failed at Source with an access error. The GitHub connection comes up as PENDING after Terraform creates it. A human must finish the handshake in the console. The error text is the same whether IAM, the source credential, or the GitHub app install is wrong. Check them in that order.
• Long WebSocket sessions were cut at 60 seconds. That is the default ALB idle timeout. Raise it above your longest session, and raise the target group deregistration delay so draining tasks do not drop live sockets.
• A web app crash-looped on staging only. Its health check got a 307 redirect from /, and staging's matcher accepted only 200-299. Point the health check at a real /health route, or accept 200-399.
• A deploy looked like a crash loop for 25 minutes, then succeeded. The app ran schema migrations at boot. An ALTER TABLE waited for locks held by the blue tasks, so the green task missed its health-check grace and was replaced, again and again. Blue served the whole time. Run migrations as a separate task before the deploy, or give the container a start period longer than the slowest migration.
• A beta stack in the production account would have deployed against production's database. The build chose the environment from the AWS account id, and two stacks shared one account. Add an explicit DEPLOY_TARGET override on the build project.
• A scheduled-job fix was reverted by the next release. The job's task definition was re-registered by hand with new settings, but the build re-registers it from the repo file on every run. Commit the change first.
• A regenerated lockfile broke every build. package-lock.json changed with no change to package.json, and npm ci failed at install. Before tagging, diff both files against the last good tag.
• A red PR check was read as a code failure. The PR build also ran a repo-wide format check that was already failing. The deploy build does not run it. Open the build log and see which phase failed before you blame the change.

Checklist

• The pipeline triggers only on this environment's tag pattern.
• buildspec.yml uses cut, not bash substrings.
• Each environment has its own task definition file, with real role ARNs, region and log group.
• The execution role can read this service's SSM path and decrypt it.
• ECR tag mutability works with the build cache.
• The health check path returns 200 and the matcher fits it.
• Prod keeps the blue termination wait until alarm rollback exists.
• After the first deploy, traffic really reaches the new task set.