Skip to main content

Backups and restore

What it is​

Copies of every database and file store, kept somewhere other than the server they came from, for a set time, and restored on purpose every quarter to prove they work.

Why we do it​

  • We found a dev box with no backups of any kind.
  • A nightly dump looked fine for weeks and was empty: the dump command failed, the pipe hid the failure, and a 20-byte compressed file passed the integrity check.
  • A client's backups were on the same server that was compromised, readable by the attacker.
  • Seven manual production snapshots containing personal data never expired.

How it fits together​

Steps​

1. Managed databases (RDS, Aurora)​

Settingprodstagingdev
Automated backup retention (PITR)14 days or more7 days1 day
Deletion protectionOnOnOff
Final snapshot on deleteOnOnOptional
Pre-migration manual snapshotEvery migrating deployYesNo
Manual snapshot expiry30 days, deleted by a job30 days7 days

Restoring never overwrites the existing database: it creates a new one. You then point the app at it or swap names.

2. Self-hosted databases: a nightly dump that checks itself​

#!/usr/bin/env bash
set -euo pipefail                      # pipefail: a failed pg_dump fails the script
F=/home/deploy/backups/acme-$(date +%F).sql.gz
docker exec acme-db pg_dump -U acme_app acme | gzip > "$F"

SIZE=$(stat -c %s "$F")
TABLES=$(zcat "$F" | grep -c '^CREATE TABLE' || true)
if [ "$SIZE" -lt 10240 ] || [ "$TABLES" -lt 20 ]; then
  echo "backup too small: $SIZE bytes, $TABLES tables" >&2
  exit 1                               # the cron wrapper alerts on non-zero exit
fi
aws s3 cp "$F" s3://acme-backups/db/ --sse aws:kms
find /home/deploy/backups -name '*.sql.gz' -mtime +3 -delete

Use the real application role name; a role that does not exist is exactly how our dump came out empty. Set the size and table thresholds from a known-good dump.

3. Files and object storage​

  • S3 buckets holding user files: versioning on, with a lifecycle rule that expires old versions after 30 to 90 days.
  • Self-hosted object storage (MinIO): replicate or sync nightly to S3.
  • Before any bulk delete or restructure, take a fresh dump and a storage snapshot. Some deletes are irreversible by design.

4. Where backups live​

A separate backup bucket, ideally in a different account (the shared or a backup account), versioned, encrypted, with Block Public Access, and only a backup role able to write. A compromised server must not be able to delete its own backups.

5. The quarterly restore drill​

  1. Pick a backup at random, not last night's.
  2. Restore it into a new, isolated database (a one-off task or a sandbox account).
  3. Check real row counts on key tables with count(*). Table statistics reset on restore, so do not trust them.
  4. Time it from start to usable. That is your real recovery time.
  5. Write the result, date and time taken in INVENTORY.md. Delete the restored copy.

If it goes wrong​

You seeIt means
Backups are tinyThe dump failed silently. Check the role name and pipefail
Restore takes far longer than expectedPlan for it: that is the downtime users will see
Snapshots pile up and cost growsNo expiry job for manual snapshots

Rules​

Rule

A backup script must fail loudly when the backup is empty or small. Check size and content, not only the exit code.

Rule

Backups are never only on the server they protect.

Rule

Manual snapshots of production data expire. Personal data kept "just in case" is a breach waiting to happen, and against our retention schedule. See Retention and deletion.

Mistakes we made​

  • A dev box had zero backups, an unencrypted disk and no alarms, found in a review.
  • A nightly dump was a 20-byte empty file for as long as nobody looked.
  • Dumps were written to /tmp on the same server.
  • Seven manual production snapshots never expired.
  • Staging was quietly pointed at near-empty load-test database clones, so testing ran against the wrong data.

Done when​

  • Managed databases: retention, deletion protection, final snapshot, pre-migration snapshots.
  • Self-hosted databases: nightly dump with size and content checks, off the box.
  • File storage versioned with lifecycle rules.
  • Backup bucket separate, versioned, encrypted.
  • Manual snapshot expiry job.
  • A restore drill this quarter, recorded with its time.
Last updated on