Upgrade Safely: Write the Rollback First

Platforms fall behind for a simple reason: every upgrade feels risky, so it gets postponed, and every postponement makes the next upgrade bigger and riskier still. The way out is to make upgrades boring. That means a template you follow every time, and the single most important line in it is the one most teams write last: how to get back.

Know a release exists before you need it

We post every new build, with its changelog, into our operations chat the day it ships.

One refinement is worth copying. Many projects publish fixes for older branches alongside the current one. A watcher that blindly announces "new release" can tempt someone into installing a back-port that is older than what is already running. We label those as "do not deploy" automatically, so nobody steps backwards by accident.

Prepare the change

Read the changelog like a contract

In the release notes, the migrations matter most: anything that runs once and cannot easily be undone, such as a schema change, decides what your rollback must include.

Pre-flight every server

Then check each server you will touch, not the fleet in general, and list its plugins, extensions and integrations against the new release.

A plugin nobody remembered is the most common reason an otherwise clean upgrade fails. Listing them per server takes minutes and removes the surprise.

Take a backup you have actually looked at

"The backup job ran" is not the same as "we have a backup". Before the change, take a consistent dump and test it:

mysqldump --single-transaction --routines --triggers --events <db> \
  | gzip > <db>-pre-upgrade-$(date +%F-%H%M).sql.gz
gzip -t <db>-pre-upgrade-*.sql.gz && ls -lh <db>-pre-upgrade-*.sql.gz

Write the rollback before the change

This is the step that turns a risky upgrade into a routine one. Before starting, write down, in order, the exact steps that would return the system to where it is now. On RPM-based systems the package side is usually short:

dnf history list | head
dnf history undo <transaction-id>

And to keep a known-good package from moving while you test, the versionlock plugin pins it:

dnf versionlock add <package>

But packages are rarely the whole story. If the release runs a schema migration, rolling back the packages alone leaves you with old code on a new schema. Your rollback then has to include restoring the dump you just took, and you need to know how long that restore takes. Write it all down, and wherever you can, rehearse it on a copy first. A rollback that exists only in your head at the moment things go wrong is not a rollback.

Change one box at a time

On redundant platforms, one node at a time. Major releases go to our own instances first and run there before any customer instance receives them.

Measure and verify

Measure the downtime instead of estimating it

"About a minute" is a guess. A probe running throughout the change gives you the real number. Start this before you begin and stop it when you are done:

while true; do
  printf '%s %s\n' "$(date +%T.%3N)" \
    "$(curl -s -o /dev/null -m 2 -w '%{http_code}' https://<service>/health)"
  sleep 0.5
done | tee upgrade-probe-$(date +%F).log

Afterwards, the first and last non-200 lines in the log bracket the interruption to within half a second. That is the number you report to the people who depend on the service, and it is the number you compare against next time. Over a quarter, these measurements are the best evidence you have that upgrades are getting safer rather than riskier.

Verify from outside, then bring the OS along

When the upgrade is done, fire a test alarm so you know monitoring still sees the service. Then use the same window to bring the operating system's packages current too. Application and foundation stay in step, and the next upgrade does not start with a pile of unrelated updates.

The template

Release noticed and announced, back-ports labelled.
Changelog read for security fixes, breaking changes and migrations.
Each server pre-flighted, including every plugin.
Fresh backup taken and inspected.
Rollback written, step by step, before the change.
One node at a time, majors on your own systems first.
Downtime measured by a probe, reported in seconds.
Verified from outside, OS updated in the same window.

If your platforms have fallen behind and the next upgrade feels like a project, we can help you get back to routine. Talk to an engineer

Write the rollback first: a template for routine upgrades