Patching and upgrades
What it is
Two different jobs. Patching applies small fixes to what you already run: OS packages, minor database versions, base images. Upgrading moves to a new major version: Node 20 to 24, PostgreSQL 12 to 16, a framework's next major. Patching is monthly and boring. Upgrades are planned, months before the old version's end of life.
Why we do it
- Old versions stop getting security fixes. One fully patched server type stopped an attack that everything else on that network let through.
- Platforms force upgrades on their schedule: AWS stops letting you create old database versions, app stores reject builds with old SDKs.
- Upstream downloads disappear. A tool's download URL started returning an error page, and our deploys failed silently for three weeks.
How it fits together
Steps
1. Keep a version table
In INVENTORY.md, one row per runtime with its end-of-life date. Check
endoflife.date when you fill it.
| Component | We run | End of life | Plan |
|---|---|---|---|
| Node.js | 24 (all APIs), 20 (two web apps) | Node 20: 30 April 2026 | Move the web apps to 24 now |
| PostgreSQL | 16 | November 2028 | Minor versions monthly |
| Base images | node:24-alpine from ECR Public | Follows Node | Rebuild monthly |
| Mobile SDKs | Flutter pinned per repo with FVM | Store deadlines | Check each quarter |
2. Monthly patching
| What | How |
|---|---|
| Self-hosted OS | Automatic security updates; reboot in the window if required (/var/run/reboot-required) |
| Base images | Rebuild every service's image monthly, even with no code change, to pick up patched layers |
| Managed databases | Allow minor version upgrades in the maintenance window; set the window at a low-traffic time |
| Dependencies | Dependabot pull requests, reviewed weekly. See GitHub organisation |
3. Major upgrades
- Read the release notes for breaking changes.
- Upgrade in dev; run the full test suite and a manual smoke test.
- Staging for at least a few days of normal use.
- Prod in the maintenance window, with a tested rollback: a database snapshot before a database upgrade, the previous image tag for an app.
For managed databases, test the major version upgrade on a snapshot restore first, and time it, so you know the real downtime.
4. Pin what you download
# wrong: follows whatever "latest" becomes, and saves an error page as the binary
curl -sL https://example-tool.io/latest/linux/tool -o tool
# right: a pinned version, fail on any HTTP error, verify the checksum
curl -fsSL https://example-tool.io/releases/1.4.2/linux/tool -o tool
echo "<sha256> tool" | sha256sum -c -
Mirror important third-party images into our own registry (ECR) with a pinned tag.
Check it worked
- The version table has no row past end of life, and none within three months without a plan.
docker imageson every self-hosted box shows images built this month.
Rules
Every download in a build or deploy script uses curl -f (fail on HTTP errors) and a pinned
version. Our deploy downloaded an HTML error page, saved it as the tool, and every later step
failed in a confusing way, unnoticed for three weeks.
Do not upgrade a global SDK on a shared machine to fix one project. Pin per project (FVM for
Flutter, .nvmrc for Node, .tool-versions).
Terraform module defaults must be versions AWS still lets you create. A default that is no longer creatable breaks every new environment.
Mistakes we made
- Deploys to one environment failed silently for three weeks after a tool's download URL was retired; its database fell five migrations behind. The production workflow had the same URL and would fail on the next release.
- Two web apps still ran Node 20, past end of life, pulled from Docker Hub, while every other service ran Node 24 from ECR Public.
- A database version in the Terraform default was no longer creatable, while prod ran a newer minor version.
- A new iOS version could not run builds from the globally installed Flutter, and the app stores required newer SDKs. The fix was per-repo pinning, not a global upgrade.
- A client's database ran a PostgreSQL version past end of life.
Done when
- Version table with end-of-life dates in the inventory.
- Monthly patching done: OS, images rebuilt, database minor versions, Dependabot.
- No component past end of life; upgrades planned three months ahead.
- Every scripted download pinned and failing on HTTP errors.