Writing a migration cutover runbook with a clock on every step
A cutover runbook gives every step a time budget, an owner, a check and a way back. How to write one, rehearse it, and decide go or no go before the night.
The difference between a migration that takes five minutes of visible downtime and one that eats a weekend is rarely the hardware. It is whether the cutover was written down as a sequence of timed, checkable, reversible steps before the night, and rehearsed until the times were known. This is how we write that document.
The problem: the night is the worst time to think
Every decision taken during the window (which sync goes first, how long to wait for a replica, when to give up and roll back) is taken by tired people under pressure, usually with the site owner on the phone. The aim of a runbook is that nobody has to decide anything during the window that could have been decided the week before.
A list of tasks is not a runbook. A runbook has five columns.
The five columns
| Step | Budget | Owner | Verify | Way back |
|---|---|---|---|---|
| Freeze writes on the source | 2 min | app | admin login refused, order queue empty | lift the freeze |
| Final file sync | 3 min | ops | rsync itemises no further changes | none needed, source untouched |
| Database final catch up and promote | 3 min | db | GTID equal on both sides, target read write | demote target, unfreeze source |
| Switch DNS and load balancer | 2 min | net | new origin answering through the public name | revert the record |
| Smoke tests | 5 min | app | checklist of pages, login, checkout, search | revert DNS |
Budget is the time the step took in rehearsal, plus a margin. Owner is a person, not a team. Verify is an observable fact, not "looks fine". Way back is the exact action that undoes the step, and if a step has no way back, it is the point of no return and must be marked as one.
Before the window: move almost everything early
The cutover should only move what changed since the last sync. Everything else happens days ahead, while the source keeps serving:
- Build the destination fresh to measured specifications, rather than restoring images of the old servers.
- Seed the database from a physical backup and keep it current by replication. On the night, catching up is a matter of seconds of delta.
- Pre synchronise files repeatedly, so the last pass is small. Do a dry run the day before to see what the final pass will move:
rsync -aHAX --numeric-ids --delete --itemize-changes --dry-run \
source:/var/www/ /var/www/ | wc -l
- Copy search indexes with the engine's own remote reindex rather than re crawling content through the application.
- Lower DNS TTLs to a few minutes at least two old TTLs before the window, so resolvers have forgotten the long value by the time you switch. Check what the world actually sees:
dig +noall +answer www.example.com @<public-resolver-1>
dig +noall +answer www.example.com @<public-resolver-2>
Rehearse on a copy, with a stopwatch
Run the whole cutover against a copy of the destination at least once, with the same commands, in order, timing each step. The rehearsal produces the budget column. It also produces the surprises: the file sync that walks millions of unchanged files slowly because of a missing flag, the replication user that lacks a privilege, the health check that points at the wrong path.
Add every lesson to the runbook template, not just to this migration's copy. That is how the tenth migration gets easier than the first.
Write the go or no go criteria in advance
Agree, before the window and with the site owner, what must be true to start, and what will cause a rollback:
- Go: replication lag on the destination below an agreed figure, last dry run sync within budget, monitoring on the destination green, the people named in the owner column present.
- Rollback trigger: any step exceeding twice its budget, or the smoke tests not passing by a fixed time after the switch.
Writing the trigger down removes the most dangerous instinct of any cutover night: "give it ten more minutes". If the clock says roll back, you roll back, and you try again on another night with a better runbook.
Make the way back cheap
A rollback that requires restoring from backup is not a rollback anyone will choose to use. Design every step so undoing it is fast:
- The source stays untouched and running until the destination has proven itself. Its data is never deleted during the window.
- On the destination, anything replaced is renamed, never removed. Going back is a rename and a DNS change.
- Database promotion happens only after both sides report the same GTID, so a demotion returns you to a known position.
- Configuration changes are applied as files you can swap, not as edits you have to remember.
mv /var/www/site /var/www/site.pre-cutover-$(date +%F)
mv /var/www/site.new /var/www/site
Run the window, then audit
During the window, one person reads the runbook aloud and records the actual time against each budget. Nobody improvises. When the last smoke test passes, keep the source running and watch both sides: traffic should drain from the old origin as caches expire. Anything still arriving at the source after the old TTL has passed is a dependency nobody listed, such as a cron job, an API client or a hard coded address, and it needs finding before the source is switched off.
Afterwards, compare the new platform with the old baseline: external exposure, response times, cache hit rate, error rates. Only then retire the source, and keep its data until the owner signs off.
Your checklist
- Every step has a budget, an owner, a check and a way back.
- Data is seeded and files pre synced days ahead; the window only moves the delta.
- TTLs lowered early and verified from outside.
- A timed rehearsal on a copy, with lessons folded into the template.
- Go and rollback criteria agreed in writing before the night.
- The source kept intact until the audit is done and the owner has signed off.
If a move is on your horizon, we are happy to help write the runbook.
Published 22 October 2026.
All field notes