The failure domain map, and the handover test that proves it

Every platform diagram has three database boxes, two load balancers and a cache pair, and every one of them is labelled "highly available". Whether it is depends on two documents most builds never produce: a map of which physical things each replica depends on, and a test sheet that records what happened when each one was taken away. Here is how we write both.

The problem: three replicas can be one failure

Redundancy counts copies. Availability depends on whether those copies can fail independently. Three database nodes on three virtual machines that the scheduler happened to place on two hosts are, for a host failure, two nodes. Three nodes on three hosts whose disks all live on one storage array are, for an array failure, none. Nothing in the application or the dashboard tells you this. You have to write it down.

Step one: list the failure domains

For a typical platform, list at least:

• Host: the physical server or blade.
• Chassis or enclosure: blades share power, cooling and interconnects.
• Storage array: and, beneath it, the storage fabric path.
• Network: top of rack switch pair, uplinks, the edge.
• Site: power feed, cooling, the building itself.

Then decide which ones the platform must survive. Surviving the loss of any one host and any one array is a sensible baseline; surviving a whole site needs a second site and is a different design.

Step two: draw the map

Make a table with one row per member of every redundant tier and one column per failure domain:

The rule is simple: within one tier, no failure domain you have chosen to survive may hold a majority of members. Read down each column for each tier. Two of three database members in enclosure 1 means an enclosure failure breaks quorum. Fix the placement, or accept the risk in writing.

Step three: get the quorum arithmetic right

If you need to survive two simultaneous host failures, as a storage layer often should, you need five members in five separate domains, not four: a fourth member adds cost without adding tolerance.

Step four: make the scheduler enforce the map

A map that depends on someone placing machines correctly will be wrong after the first maintenance. Make the platform enforce it.

On a virtualization manager such as OLVM, create a negative, enforcing affinity group per tier, so two members can never run on the same host, even after a host is drained.

On Kubernetes, use required pod anti-affinity and a PodDisruptionBudget for every quorum workload:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      [ { labelSelector: { matchLabels: { app: galera } },
          topologyKey: kubernetes.io/hostname } ]

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: galera }
spec:
  minAvailable: 2
  selector:
    matchLabels: { app: galera }

The anti-affinity keeps members on different nodes; label nodes by chassis and add a second rule on that label if enclosures are a domain. The budget means a rolling node update can never evict a second member while one is already down. Check the result, not the intent:

kubectl get pods -l app=galera -o wide

Step five: the handover test sheet

Now prove it. For each tier, take away one member the way a real failure would, with the application connected and real traffic flowing, and record what happened:

Run a timestamped probe from outside during every test, so "what clients saw" is a measurement:

while true; do
  printf '%s ' "$(date +%T.%N)"
  curl -s -o /dev/null -w '%{http_code} %{time_total}\n' https://app.example.com/health
  sleep 0.2
done | tee failover-$(date +%F).log

Then restore each member and record that it rejoined cleanly, as a replica of the new primary rather than a rival to it. A test that ends with two primaries has found the most important bug on the sheet.

Step six: have someone else verify it

The person who built a platform reads the build notes when checking it, and the build notes say what was intended. Have a second engineer verify the map against the live platform from a different machine: query the virtualization manager for actual placement, the storage for actual disk locations, the cluster for actual membership. Differences between the map and reality are the findings that matter.

Your checklist

• The failure domains you will survive, written down.
• A map of every member of every tier against those domains, with no majority in one domain.
• Quorum sizes chosen by arithmetic, odd numbers only.
• Anti-affinity and disruption budgets enforced by the platform, and checked.
• A handover test sheet with measured times and client side results for every tier.
• Independent verification against the live platform, not the notes.

If you would like a second opinion on how independent your redundancy really is, we would be glad to map it with you.

Talk to an engineer

Related: High-availability platform builds · Clustered databases with Percona XtraDB Cluster · Object caching with automatic failover