Spared no expense

Out-of-band management on every machine

We do not use IPMI.

Every Dell server we run carries iDRAC Enterprise with Datacenter licensing. Every HPE server carries iLO Advanced. Every blade chassis is run through its Onboard Administrator. These are licensed, paid-for management controllers, not the stripped IPMI port that comes free on a white-box machine. The difference is the difference between a server we can operate and a server somebody has to go and visit.

A lights-out controller is a computer in its own right, with its own processor, firmware, network port and power path, depending on none of the operating system, the kernel, the production network cards or the disks. A host at a kernel panic, half-booted, sitting in its firmware with no operating system at all, or switched off entirely, is still reachable and still reporting. From it we get a full remote console, virtual media so a machine can be rebuilt from nothing across the network, power control, firmware and BIOS management, and every hardware sensor. On the blade chassis the Onboard Administrator reaches every controller in the enclosure across the backplane, and is itself a redundant pair with automatic failover.

In practice. When a rack of blade chassis needed iLO Advanced licensed, and a management account created on every blade in it, we did not log in to a single blade. We pushed it through each chassis's Onboard Administrator and the rack was finished in minutes, including blades racked that week that had no credentials of their own yet. It is a runbook.

The management network is not the production network

Customer traffic and management traffic share nothing in our datacenters. Not a port, not a switch, not an address, not a route.

The controller has its own physical port and its own cable to its own switching. We do not partition a production network card and let the controller borrow a slice of it, which is the usual shortcut and the usual reason management ends up one mistake away from the internet. Management lives on its own switches, or on its own VLANs on fabrics that carry nothing else, in private address space that is not routed to or from any customer network and is never announced to the internet.

We call it an air gap because that is how it behaves. A customer virtual machine, even one fully compromised, scanning at line rate for a week, cannot see an iLO or an iDRAC at all. Not refused, not filtered: absent. There is nowhere for the packets to go. It holds in the other direction too, and that matters just as much: no management traffic rides a customer VLAN, so our console sessions, monitoring and backups never enter a customer's broadcast domain or compete for their bandwidth. The isolation is real enough to be inconvenient to us, which is the proof that it is real: a controller on a management VLAN cannot casually reach an internal web server to pull a firmware image, because nothing is permitted that was not asked for and written down. The engineer takes the extra step.

In practice. One canonical management access list is bound on every platform we run, on Cisco Nexus login lines, per routing instance on the routers, per interface for SSH and HTTPS on the firewalls, permitting only named management subnets, with control plane policing rate-limiting whatever is left in hardware. Then we check it the way an attacker would, not the way a documentation review would: from outside our own network, scanning our own public address space, confirming nothing answers.

The management plane itself

The management plane is a network in its own right and we built it like one. It runs on our own encrypted multi-site backbone: Cisco DMVPN, IPsec at AES-256, BGP carried over the tunnels, joining Athens, Amsterdam and Toronto to a management hub in Cyprus that is deliberately not in any of the buildings it manages. The hubs are redundant, with equal-cost paths across them and across diverse circuits, and failure detection that notices a dead path in well under a second.

Staff reach it one way: through VPN gateways and firewalls we own and operate ourselves, on OPNsense and Cisco Firepower, then through jump hosts inside the plane rather than straight from an engineer's laptop. Different trust tiers never share a firewall object, so widening one group's access cannot silently widen another's. When we work on a firewall we keep an independent out-of-band path into that site, because an engineer changing a firewall will cut off their own access at least once.

Toward the backup and disaster recovery planes the policy is deny by default and the data flows one way. Production never pushes to backup. The backup side pulls. That is enforced with explicit deny rules on the firewall appliances, not with a host firewall a compromised host could edit, and the backup segments have no route to the internet at all. Only the management range reaches them. If a production machine is taken over completely, the attacker gains no knowledge of the backup infrastructure and no path to it. From where they are standing, it does not exist.

Changes follow a fixed discipline. Every configuration is backed up before it is touched, to the device and to disk, and the backup is checked before we proceed. One command at a time, verified live against the running system rather than by re-reading what we just applied, with a stated one-step rollback. On a redundant pair, one half at a time, sessions confirmed back up before the other is touched. That is why we can work on the backbone in the middle of a working day.

In practice. We moved a customer's entire site-to-site VPN estate from one country to another in a single session, ahead of a fixed deadline. The hub was cloned, carried across and brought up in parallel with the original, so both were live at once and there was always somewhere for the tunnels to be. The branch routers were moved across one at a time, each verified up before the next was touched, each failing over cleanly as it went. Nobody travelled to a branch office.

The hardening baseline on every host

Every machine we build starts from the same baseline, and the baseline is not up for negotiation. SELinux is enforcing, and we do not switch it off to make an application cooperate, we write the policy instead. Every host runs its own firewall as well as the perimeter. Filesystems are XFS. Access is by SSH key, with strict host key checking and bounded session timeouts.

Virtual machines have no swap, and that is deliberate. A virtual machine's disk lives on an enterprise all-flash array, so swap means spending that array's endurance as overflow memory for an application that has outgrown its allocation. A machine that needs more memory gets more memory.

Patching depends on nobody else. We run our own package mirror in our own datacenter, carrying the repositories our fleet installs from, upstream signatures intact, reachable only from our own address ranges. A public mirror having a bad day is not an event here, and a server deliberately unable to talk to the internet is still a server we can fully patch.

In practice. When a critical vulnerability in a widely used runtime was published, the fixed packages were in production on every affected server the same day, before the end of the same night. Not by hand and not in a rush: the rollout takes a backup, upgrades one host at a time, samples real sites on that host for a healthy response before and after, and halts the whole sweep automatically if one does not come back correctly. Fast and careful are not opposites if you build the thing that does it for you.

Monitoring that watches the watchers

All of this is worth nothing if a failure can be quiet, so we spend real effort making sure it cannot be. Every device ships its logs to one central platform: routers, switches, firewalls, storage arrays, chassis controllers, hosts. Sources are normalised to clean device names on the way in, and logs land in purpose-built streams per role rather than one undifferentiated firehose. The serious severities go straight to the operations chat system where the on-call engineer already is, reaching a human in seconds, and we know it is seconds because we have timed it with a real alarm rather than reading the configuration back to ourselves. Alarms exist for what actually precedes an outage: a routing peer lost, a link down, encryption failures on the backbone, hardware and power faults, storage fabric errors. At the edge, dedicated sensors read mirrored traffic off the border routers and watch continuously for volumetric and denial of service patterns, so an attack is something we are already measuring rather than something a customer tells us about.

A device can look perfectly healthy from its own command line while silently sending nothing to the collector, so "logging is configured" is not evidence here. We confirm the log arrives, at the far end, in the stream it belongs to, after every change that could have disturbed it. The monitoring and detection systems have their own watchdogs on separate machines, so "the thing that was watching has stopped watching" is itself an alarm. Silence is never good news.

In practice. Our routine read-only health sweep checks the obvious things, environmentals, routing and session state, redundancy consistency, interface errors, and then two more: whether each device is still genuinely shipping its logs, and whether it is still genuinely getting its configuration backed up. Those two are in the sweep because they are the failures that hide the others.

A CTO evaluating us is welcome to ask for the access list, the runbook and the alarm routing. We will show you all three.