Maintenance windows: what gets done under load instead

Two kinds of planned work

Planned work falls into two groups. The first is absorbed by the design: a storage controller taking new firmware, a leaf switch being replaced, a hypervisor host going down for memory, a firewall pair rolling an update. The second changes something that has no partner at that moment: a platform cutover, the final synchronisation of a storage migration, an irreversible schema change. The first group is done under load. The second gets a window, and the window is designed rather than merely announced.

What the design absorbs

An HPE 3PAR array carries two controller nodes serving the same volumes, with every host attached through two independent Brocade fabrics. Firmware goes on one node at a time; while it restarts, the hosts' multipath layer keeps the I/O flowing over the surviving paths, and the node rejoins before the second one is touched. The same pattern applies at every layer that was built in pairs. Cisco Nexus 9000 leaves are deployed in pairs with each host bonded across both, so a switch can take an upgrade or a replacement while the other link carries the traffic. Hypervisor hosts are emptied by live migration before anything is done to them, and the virtual machines move back afterwards. Edge routers hold sessions with more than one carrier, and a session is drained before its router is touched.

None of this is unusual in a serious datacenter. What matters is whether the pairs are exercised. A partner that exists on the diagram but has not carried production is a guess, and the moment to discover that it was a guess is not during the change.

Verify the partner before touching the primary

The useful discipline is the check that happens before the change. Before a controller is taken down, every attached host is asked how many active paths it holds to the array; a host reporting a single path is fixed first, because for that host the window would otherwise arrive uninvited. Before a switch is touched, the bond on each host is read to confirm both members are up and passing traffic, and the second switch is confirmed to be forwarding rather than merely powered. Before a host is drained, the migration target is confirmed to have the headroom for what is coming.

# on each attached host, before controller work: every LUN should show two or more active paths
$ multipath -ll | grep -c 'active ready'

# before switch work: both bond members up and passing traffic
$ grep -E 'Slave Interface|MII Status' /proc/net/bonding/bond0

Then the change is made, and the same readings are taken again as the partner rejoins. The result is planned work that does not interrupt you, which is different from planned work that is not communicated: the change is in our log and, where it touches your platform, in your inbox, with the mechanism stated.

The changes that deserve a window

A window is the right answer when a step is irreversible, when the component by its nature has no partner at that moment, or when the vendor's procedure requires the service to be quiesced. A platform cutover is the common case: the moment traffic moves from the old environment to the new one, there is a span, usually bounded by DNS propagation or a VIP move, in which the service is in transit.

A well-designed window is recognisable. It names the component and the mechanism, so you know what is changing and why it cannot be done under load. It states the expected customer-visible impact separately from the length of the operation; a two-hour operation can have a visible impact of minutes, and the notice says which is which. It has a decision point and a rollback that was rehearsed, not described. It is agreed with you: if a change could touch your traffic, we agree the timing with you first. And it ends with a report that compares the actual to the announced.

Reading a maintenance notice

Three questions tell you most of what you need about the design under any platform, ours included: what is the component, why can it not be done under load, and what is the expected visible impact against the duration of the operation. A notice that answers all three is a plan. We write ours to answer them.