Skip to main content

Scheduled jobs

What it is​

Work that runs on a timer instead of a request: nightly purges, reminder batches, reconciliation sweeps, backups. On AWS we run them as EventBridge Scheduler tasks on ECS. On a self-hosted box, as a systemd timer (preferred over cron, because it logs and can alert).

Why we do it carefully​

A scheduled job fails quietly. Nobody is waiting for its response. Ours have been cut off by a timeout, overwritten by an unrelated deploy, run twice at once, and failed silently on a missing permission.

How it fits together​

Steps​

  1. One job, one task definition, in the repo, deployed by the pipeline like any service.
  2. A lock so two runs never overlap: a database advisory lock, or a Redis key with a timeout.
  3. A timeout longer than the slowest normal run, and shorter than the schedule interval.
  4. Guards for destructive jobs. A purge checks it is about to delete a plausible number of rows, and stops if the number is far larger than usual.
  5. A metric per run (success, failure, duration, items processed), with alarms on failure and on "did not run".
  6. Skip boot-time work. If the app normally runs migrations at start-up, the scheduled task must not.

Rules​

Rule

Never fix a scheduled task by registering a new revision by hand. The next build registers it again from the repo file and silently undoes the fix. Change the file.

Rule

Every destructive job has a guard and a dry-run mode, and logs what it would delete before deleting.

Mistakes we made​

  • A reminder batch was cut off at 30 seconds by a timeout, and the hand-registered fix would have been overwritten by the next build.
  • A sweeper could overlap itself when one run took longer than the interval.
  • A purge's file deletes failed silently for lack of a permission, while its database deletes succeeded.
  • A new task revision was picked up by the scheduler only after about a minute, so a test run straight after deploy used the old one.

Done when​

  • Each job in the repo, with a lock, a timeout and a metric.
  • Destructive jobs have guards and dry-run.
  • Alarms on failure and on missed runs.
Last updated on