Skip to main content

Self-hosted server baseline

What it is​

The minimum setup for any server we run ourselves: a VPS at a hosting provider, or an EC2 box. Users and access, firewall, folder layout, Docker Compose settings, the reverse proxy, automatic patches and monitoring.

Why we do it​

Our self-hosted boxes had no health checks, no log limits, no memory limits, no backups, no alarms, passwords in text files and, in one case, a database admin tool open to the internet. Each was a reasonable shortcut on the day it was taken.

Before you start​

  • A reason the service cannot run on Fargate or a managed service. Write it in the inventory.
  • The box is in the inventory with an owner.

How it fits together​

Steps​

1. Access​

ServerAccess
EC2SSM Session Manager only. No inbound security group rules, no key pair. Copy files through S3
VPSSSH with keys only (no passwords), on a VPN or an allowlist, never open to the internet. Root login off

Two OS users: an admin user for people, a deploy user for automation with only the rights it needs. Never deploy as root.

# VPS: SSH hardening in /etc/ssh/sshd_config
PasswordAuthentication no
PermitRootLogin no

2. Firewall​

Only 80 and 443 open to the world, for the reverse proxy. Everything else (database, admin tools, metrics) listens on 127.0.0.1 or the internal Docker network.

sudo ufw default deny incoming
sudo ufw allow 80,443/tcp
sudo ufw allow from <vpn-range> to any port 22
sudo ufw enable

3. Folder layout​

/etc/docker/services/        shared services: proxy, database
/home/deploy/apps/<app>/     one folder per app: compose file, .env (mode 600)
/home/deploy/backups/        local staging for backups before upload

4. Docker Compose settings every service needs​

services:
  api:
    image: registry.example.com/acme-api:1.4.0   # a pinned tag, never "latest"
    restart: unless-stopped
    mem_limit: 768m
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
    logging:
      driver: json-file
      options: { max-size: "20m", max-file: "5" }
    networks: [internal]
SettingWithout it
mem_limitOne container can starve the whole box, including the SSM agent or SSH
healthcheckA hung container counts as "up"
logging limitsLogs grow until the disk is full
Pinned image tagA restart pulls a different version

5. Reverse proxy and TLS​

Traefik or Caddy get certificates from Let's Encrypt automatically. nginx needs Certbot and its renewal timer. Redirect HTTP to HTTPS. See Disks, logs and certificates for checking renewal.

6. Automatic security patches​

# Ubuntu / Debian
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades
# Amazon Linux 2023
sudo dnf install -y dnf-automatic
sudo sed -i 's/^apply_updates.*/apply_updates = yes/' /etc/dnf/automatic.conf
sudo systemctl enable --now dnf-automatic.timer

Very small instances (512 MB) can run out of memory during a full update. Use at least 1 GB for any box that patches itself.

7. Monitoring and backups​

  • An uptime check on the public health URL, and alarms on disk above 80% and memory above 90% (CloudWatch agent on EC2; the provider's monitoring or node-exporter on a VPS).
  • Nightly backups off the box. See Backups and restore.

8. Deploying​

Build images in CI and push them to a registry; the server only pulls and restarts:

docker compose pull && docker compose up -d && docker image prune -f

Check it worked​

sudo ss -tlnp            # only the proxy on 0.0.0.0:80 and :443
docker ps --format '{{.Names}}\t{{.Status}}'   # every container "healthy"
sudo unattended-upgrade --dry-run

From a laptop outside the VPN, try the database port: it must not answer.

If it goes wrong​

You seeIt means
The box stops answering SSM or SSH during a buildMemory ran out. Build in CI; set memory limits
git pull on the server fails with an auth errorThe remote URL holds an expired token. Deploy images, not git checkouts
A container is "unhealthy" but still servingThe health check path is wrong

Rules​

Rule

No admin tool is published on a public port: database consoles, queue dashboards, password managers. Reach them through SSM or the VPN.

Rule

Never git pull and build on a production server. It needs credentials on the server, memory you do not have, and it deploys whatever the branch happens to contain.

Done when​

  • SSM-only (EC2) or key-only SSH behind the VPN (VPS); root login off.
  • Only 80 and 443 public.
  • Every service has a memory limit, health check, log limits and a pinned image.
  • TLS automatic and redirecting.
  • Automatic security patches on.
  • Uptime, disk and memory alarms; nightly off-box backups.
Last updated on