The scheduled job that failed silently for weeks

A nightly job — a backup, an export, an invoice sync — had been quietly not running for weeks. Not erroring. Not running. The OAuth refresh token it used to talk to a Google or Microsoft API had expired, the job died before it did anything, and no error was ever raised because the code never got far enough to raise one. The first sign anything was wrong came when someone went looking for data that should have been there and wasn't.

This is the blind spot in almost every cron setup: your alerting is wired to errors during execution. If the job never executes, or dies at the first line, there is no error to catch. Silence looks exactly like success.

You're monitoring the wrong event

The pattern is called a dead man's switch, or a heartbeat check. The last line of your job — after everything has actually completed — sends a signal. An external service expects that signal on a schedule and yells when it's late.

Wire up a heartbeat this afternoon

Healthchecks.io is the common pick — open source, has a free tier, and you can self-host it later if you want. Cronitor and Dead Man's Snitch do the same job. Check each vendor's current pricing before you commit; the free tiers cover a handful of checks.

1. Create a check in the service and set its expected period and a grace window — say, runs daily, alert if no ping within 1 hour of the deadline.
2. Copy the unique ping URL it gives you.
3. Add the ping as the last action in your job, only reached on success. For a shell script:

   0 2 * * * /path/backup.sh && curl -fsS -m 10 --retry 3 https://hc-ping.com/YOUR-UUID

   The && matters — the ping only fires if backup.sh exits cleanly.
4. For a job that dies mid-run, ping the URL from inside the code at the end, or use the service's start/fail signals so a crash reports a failure explicitly.
5. Set the alert destination to email plus SMS or a chat channel — somewhere you'll actually notice.

If the machine reboots, the network drops, or the token dies, no ping arrives — and now that's an event, not a gap in your data.

Fix the token, not just the alarm

The heartbeat tells you something broke; the OAuth token is why. If you use Google APIs, a refresh token issued while your app is in Testing publishing status expires after about seven days — move the app to In production so refresh tokens stop expiring on that clock. Tokens also die when a password changes, access is revoked, or the account is inactive too long. Store the token where you'll see it, and let the heartbeat cover the rest.

Where DIY stops

A heartbeat per job is easy. What's harder is a job that half-runs — pings success but processed zero rows, or wrote a truncated file. For that you need the check to assert on output (row counts, file size, a checksum), and that's real code plus judgement about what "healthy" means for each job. If you're running more than a few interdependent jobs, or handling money and tax records, get someone to design the monitoring around the actual failure modes rather than bolting pings onto everything.