Server and maintenance: read this first
Launching a system is the start of the work, not the end. Servers need patches, backups that really restore, upgrades before a version dies, certificates that renew, disks that do not fill, and old resources switched off. This module is how we do that, for every kind of server we run. Read it before you take over any environment.
The three kinds of server
| Kind | Who patches the OS | Who backs up | What we still own |
|---|---|---|---|
| Managed (Fargate, RDS, Aurora, ElastiCache) | AWS | AWS, if we turn it on and set retention | Versions, sizes, backups settings, restore drills, cost |
| Self-hosted (a VPS, an EC2 box) | Us | Us | Everything: OS, Docker, TLS, disk, backups, monitoring, access |
| Client on-premise | The client, unless the contract says us | The client | Advice, and our application's own settings |
Prefer managed. Every self-hosted box is a list of chores someone must remember. Most of our maintenance incidents came from self-hosted boxes.
Words you will meet
| Word | Plain meaning |
|---|---|
| Patch | An update that fixes bugs or security holes, usually small |
| EOL (end of life) | The date a version stops getting security fixes |
| Backup vs snapshot | A backup is any copy you can restore. A snapshot is a point-in-time copy of a disk or database |
| PITR | Point-in-time recovery: restore a database to any second in the last N days |
| Restore drill | Actually restoring a backup somewhere safe, to prove it works |
| Rightsizing | Choosing the smallest size that safely does the job |
| Decommission | Archiving, then switching off and deleting something no longer used |
The pages in this module
| Page | Read it when |
|---|---|
| Server inventory | You start on a project, or create any server |
| Self-hosted server baseline | You set up a VPS or EC2 box |
| Patching and upgrades | Every month, and when a version nears end of life |
| Backups and restore | You add a database or storage, and every quarter |
| Disks, logs and certificates | A disk fills, or you set up TLS |
| Scheduled jobs | You add a cron job or a scheduled task |
| Cost and clean-up | Every month, and when something is retired |
| Maintenance calendar | Always: the daily, weekly, monthly and quarterly checklist |
| Server mistakes | Your first week |
The daily production check and incident handling are in DevOps. This module is about keeping things healthy between incidents.
Rules
Every server is in the inventory with an owner. A server nobody owns is a server nobody patches.
A backup you have not restored is a hope, not a backup. Restore every backup type at least once a quarter.
Never build on a production server. Build in CI, ship an image or a bundle, and run it. A build on a small server once used all its memory and locked us out for 25 minutes.
Done when
- You know which servers your project has, and which kind each is.
- You have read the maintenance calendar.