Making a backend restart invisible to your visitors

Having two web servers behind a load balancer does not mean a restart is invisible. It means a restart produces a short burst of errors instead of an outage. The difference between "short" and "none" lives in a few settings at the edge, and in testing them with real traffic before you rely on them. This is how we close that gap.

Why redundancy alone still shows errors

When PHP-FPM or nginx restarts on one web node, three things happen in the second or two it takes. Requests already in flight on that node are cut. New requests keep arriving, because the load balancer has not yet noticed the node is gone. And when the node comes back, it may be sent a full share of traffic before its caches and opcode cache are warm.

A load balancer with default settings handles none of these gracefully. Its health check runs every few seconds and needs several failures before it marks a node down, so for that window it is deliberately sending visitors to a server that is not answering. Each of those requests becomes a 502 or 503 on someone's screen.

Retry and redispatch at the edge

The first fix is to let HAProxy try again, somewhere else, when a connection fails:

defaults
    mode http
    retries 3
    option redispatch 1
    retry-on conn-failure empty-response response-timeout
    timeout connect 3s

retries lets HAProxy retry a request whose connection failed. option redispatch 1 makes the retry go to a different server instead of hammering the one that just refused. retry-on extends retries beyond refused connections to empty responses and timeouts, which is what a process mid restart often produces.

Retries are safe for requests that have not been processed. For POST requests that may have reached the application, HAProxy is conservative by default, and that is what you want: a checkout should never be submitted twice.

Health checks that see the application, not the port

A TCP check passes as long as nginx is listening, even when every PHP worker is wedged. Check something that exercises the stack:

backend be_web
    option httpchk GET /healthz
    http-check expect status 200
    default-server inter 2s fall 3 rise 2 slowstart 20s
    server web1 <web1>:80 check
    server web2 <web2>:80 check

The /healthz endpoint should be a tiny PHP script that touches what matters (opens a database connection, reads one cache key) and returns quickly. fall 3 means one slow check during a log rotation or a reload does not evict a healthy node; rise 2 keeps a node out until it answers twice. slowstart ramps a returning server up gradually, so a node with cold caches is not handed a full share of traffic at once.

Drain before you restart

The cleanest restart is one the load balancer knows about in advance. HAProxy's runtime API lets you take a node out of rotation without dropping established requests:

echo "set server be_web/web1 state drain" | socat stdio /run/haproxy/admin.sock
# wait for current sessions to finish
echo "show servers state be_web" | socat stdio /run/haproxy/admin.sock
systemctl restart php-fpm        # on web1
echo "set server be_web/web1 state ready" | socat stdio /run/haproxy/admin.sock

Wrap that in a script and every planned restart (a PHP security update, a configuration change, a kernel reboot) becomes drain, wait, restart, verify, return, one node at a time.

Let the cache cover the gap

A full page cache does more than speed up anonymous traffic. With grace configured, Varnish keeps serving the last good copy of a page when it cannot fetch a fresh one:

import std;

sub vcl_recv {
    if (std.healthy(req.backend_hint)) {
        set req.grace = 10s;
    }
}

sub vcl_backend_response {
    set beresp.ttl = 5m;
    set beresp.grace = 24h;
}

While backends are healthy, objects are refreshed within seconds of expiry. When every backend is unhealthy, Varnish may serve objects up to a day past their TTL. Readers keep getting pages during a database maintenance or a wedged PHP pool, and the logged in and checkout paths, which bypass the cache by design, are the only ones that depend on the backends being up.

The edge itself needs a partner

Two edge nodes running keepalived share floating addresses, and a tracking script demotes an edge whose HAProxy has stopped:

vrrp_script chk_haproxy {
    script "/usr/sbin/pidof haproxy"
    interval 2
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    track_script { chk_haproxy }
    virtual_ipaddress { <floating address>/<prefix> }
}

We build edges criss-cross: each node is primary for one public address and standby for the other, with the DNS record carrying both. Both edges carry traffic all the time, and a failure moves one address to the survivor within seconds.

Prove it with traffic, not with a feeling

A restart is only invisible if you have watched it be invisible. The test we run before handover is simple:

1. Put steady load on the site from outside, with a mix of cached and uncached URLs. A small load generator or a curl loop is enough.
2. Count responses by status code at the client, timestamped.
3. Restart PHP-FPM on one web node. Then the other. Then fail an edge.
4. Compare the client side count with the origin logs.

while true; do
  printf '%s ' "$(date +%T.%N)"
  curl -s -o /dev/null -w '%{http_code} %{time_total}\n' https://www.example.com/
  sleep 0.2
done | tee restart-test.log
grep -vc ' 200 ' restart-test.log

The origin will usually log a handful of errors during each restart: that is the gap the edge is meant to absorb. What matters is the number on the client side. When retry, redispatch and grace are working, the visitor count stays at nothing while the origin shows its brief restart gap. If any errors do reach the client, the timestamps tell you which layer let them through.

Your checklist

• HAProxy retries with redispatch, and retry-on covering empty responses and timeouts.
• An HTTP health check that touches PHP and the database, with fall and rise tuned against flapping.
• Planned restarts that drain first, through the runtime API.
• Varnish grace long enough to cover your longest backend maintenance.
• Two edges with floating addresses and a tracking script, both carrying traffic.
• A restart test run under load, with the client side error count written down.

If you would like help tuning a stack until its restarts disappear, we do this every week.

Talk to an engineer

Related: High-availability web tiers, N+1 at every layer · Object caching with automatic failover · Edge traffic control for crawlers and scrapers