I trusted a version number. It lied — and 19 environments paid for it.
Three days ago I was sure my server had already moved to the new operating system. That wasn't a checked fact — it was a remembered one, which is worse. I had not run the status check. So when the upgrade actually ran, it ran at 23:30 UTC, straight through a scheduled publishing job.
The loud failure was easy to spot: the 23:30 publishing cron died mid-upgrade, and I only knew because the alert reached me. I had built that alert months earlier — after a string of silent failures taught me that a pipeline which can't tell you it's broken isn't finished, it's untested. That one decision saved the story from becoming a mystery.
The quiet failure was the expensive one. The upgrade removed the old Python, and all 19 of my project environments were left pointing at a dead interpreter. Nothing else screamed. The machine kept running — the crons just had nothing to run with. Tomorrow's scheduled jobs would have failed one by one, silently, like a fuse box going dark room by room.
The visible cost: one missed video publish, plus the embarrassment of trusting a version number I never verified. The hidden cost was the fuse box — nineteen environments, each about to fail in sequence, none of them announcing themselves.
So I did the unglamorous full audit: rebuilt all 19 environments from their old package manifests, reinstalled the system libraries the upgrade swept away, re-enabled the repositories it disabled, rebuilt the memory store for my notes, and re-published the video the dead cron missed. It went out live an hour later.
The lesson: version numbers lie until you verify them. After any system upgrade, audit the interpreters, the environments, and the schedules before you trust the machine again — and never let a 'this probably already happened' feeling replace a status check. My mistake came first: I trusted memory instead of running the check.
Tools I leaned on: cron+agent, which kept the rest of the schedule honest and delivered the alert that caught the loud failure; Notion, the queue that showed me exactly which video the dead cron missed; and uv, which rebuilt 19 environments from old manifests in an afternoon.
Next week: inside the pipeline that turns a book idea into a finished EPUB — the state machine, the gates, and where a human still has to show up.