How to acceptance-test a Kubernetes cluster before production

A freshly installed OpenShift or OKD cluster greets you with a console full of green. Every operator reports available, every node is ready, and it is tempting to call it done. Green tells you the components started. It does not tell you that a volume can be restored, that traffic survives losing a node, or that one tenant cannot read another tenant's secrets. Those you have to test, on purpose, before anything you care about runs there.

Status is not behaviour

Cluster health checks are built to report whether a component is running. Production depends on how the platform behaves when something goes wrong: a node dies, a disk fills, a rolling update drains a node carrying a database. The only way to know is to cause those conditions in a controlled way and watch. The test plan below covers the layers that matter most. Adapt it, but write yours down, run every step by hand the first time, and keep the evidence.

Storage: create, write, grow, snapshot, restore, compare

Stateful workloads stand or fall on storage. For each storage class you intend to offer:

oc apply -f test-pvc.yaml
oc rsh <test-pod> sh -c 'dd if=/dev/urandom of=/data/blob bs=1M count=256 && sha256sum /data/blob'
oc patch pvc test-pvc -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
oc apply -f test-snapshot.yaml
oc apply -f restore-pvc-from-snapshot.yaml
oc rsh <restore-pod> sha256sum /data/blob

The checksums must match. Confirm the expansion is visible inside the pod with df, not just in the claim. For shared read-write-many storage, mount the same claim from pods on two different nodes and confirm both see each other's writes. Then check what happens when a claim is deleted: production data should sit on a storage class whose reclaim policy keeps the volume, so a mistaken delete does not destroy it.

Failure: kill a node and time the recovery

Pick the node carrying live ingress traffic and power it off from its management controller, not with a polite shutdown. From outside the cluster, run a loop that timestamps every request:

while true; do
  printf '%s ' "$(date +%T.%N)"
  curl -s -o /dev/null -w '%{http_code}\n' --max-time 2 https://test-app.example.com/
  sleep 0.2
done

The output shows exactly how long the service was unavailable, which is the number your users would feel. Then watch what the orchestrator does: how long until the node is marked not ready, whether pods are rescheduled, whether stateful pods come back with their data. Power the node back on and confirm it rejoins cleanly. How the network learns about the failure matters here. If the service address is announced from every node over BGP and the network side runs fast failure detection such as BFD, traffic can move away from a dead node in seconds, well before Kubernetes itself has decided the node is gone.

Maintenance: prove disruption budgets hold quorum

Rolling node maintenance and platform updates drain nodes one after another. A clustered database needs a majority of members up at all times, and a PodDisruptionBudget is how you tell the platform so:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: galera-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: galera

Then test it. With a three-member cluster and one member already down (its node drained, so only two members are running), try to drain a second member's node:

oc adm cordon <node-a>
oc adm drain <node-b> --ignore-daemonsets --delete-emptydir-data
oc get pdb

The drain should wait rather than evict, because evicting would break quorum. If it evicts, the budget is wrong or missing, and you have just found out in a test instead of during an update. Spread the members across nodes with topology spread constraints or anti-affinity, so one node failure can never take two members.

Builds and the registry

Confirm the cluster can build and run its own images, because an internal registry that cannot accept a push will only be discovered on a deploy day:

oc new-build --name=smoke --binary --strategy=docker
oc start-build smoke --from-dir=. --follow
oc new-app smoke

Access: test what a tenant cannot do

Tenancy is defined by what a user is refused. Create a scoped test user per tenant and check the boundaries explicitly:

oc auth can-i get secrets -n tenant-b --as=tenant-a-user
oc auth can-i create pods -n tenant-a --as=tenant-a-user
oc auth can-i list nodes --as=tenant-a-user

The first and third should say no, the second yes. Check that quotas stop one tenant from consuming the cluster, that each tenant's ingress serves only its own domains with a valid certificate, and that traffic leaving the cluster toward your other systems comes from the egress identity your firewall expects.

Keep the evidence

Record each test with its command, output and result. When something changes later, a platform update or a new storage class, rerun the relevant section and compare. A cluster that passed its tests once and was never tested again slowly becomes an untested cluster.

How we do it

We build OpenShift and OKD on dedicated HPE servers, with persistent block and shared storage inside the cluster, public service addresses announced from the nodes over BGP, and per-tenant ingress and certificates. Before a cluster carries a customer workload it goes through more than thirty hands-on checks of this kind, from byte-for-byte storage restores to forced failures to access scoping. In one acceptance test we powered off the node carrying live traffic, and the network rerouted in about three seconds, long before the orchestrator declared the node gone. Clustered databases on the platform are built to the standard above: disruption budgets that keep quorum, storage that is retained, members spread across nodes.

If you are about to put production on a new cluster and have only seen it green, we are glad to help you test it properly first.

Talk to an engineer

Related: Private Kubernetes on bare metal, OpenShift and OKD · Bare metal servers on HPE and Dell · Private cloud on KVM with OLVM