Instant 503 or 403: check above the app first
An error page that arrives with no latency was served before your application ran, so the checklist starts at the edge and works down to the code last.
In this article 4 sections
- A slow error gives you something to profile. An instant one (a 503, 502, 403 or 429 that arrives in a few milliseconds) means a layer above the application answered, and that layer is where the checklist starts.
- The layers above the application, top down: CDN and edge rules, the load balancer and its backend state, the control panel or maintenance switch, web-server rewrite and deny rules, DNS, filesystem permissions and SELinux.
- Identify the layer from the response itself: server headers, body shape, time to first byte, and the same request repeated directly against the origin, bypassing one layer at a time.
No latency is the first measurement
Time to first byte tells you whether the application did any work. An error that arrives at about the network round-trip time was decided without it. An error the application generated carries the time it spent before giving up.
curl -s -o /dev/null -w 'code=%{http_code} ttfb=%{time_starttransfer} total=%{time_total}\n' https://site.example/
curl -sI https://site.example/The headers name the layer more often than not. A Server header or a Via header that belongs to the edge or the balancer rather than the web server, an X-Cache or an edge request id, a plain-text body where the application would render a template: each says who answered. With that in hand the checklist is short.
The checklist, top down
The CDN or edge. A WAF rule, a rate limit, a geo rule, a bot challenge, or a certificate problem between the edge and the origin, which the edge reports as its own 5xx. Bypass it: resolve the name straight to the origin and repeat the request.
curl -s -o /dev/null -w 'code=%{http_code} ttfb=%{time_starttransfer}\n' \
--resolve site.example:443:ORIGIN_IP https://site.example/Origin answers 200, edge answers 5xx: the edge owns the error, and so does the rule that produced it.
The load balancer. HAProxy answers 503 by itself when no backend is up, with its own Server header and a one-line body. That is a health-check question, not an application one: a check path that changed in the last deploy, a check port that moved, or a backend marked down for a reason in the stats page. Request a backend directly from the balancer host to confirm it serves.
The platform layer. A suspension or maintenance state in the control panel, a deploy tool's maintenance switch, a service scaled to zero. Each serves a fixed page without invoking the application and each is a flag you can read, not a theory.
Web-server rules. A single return, rewrite or deny directive answers instantly and looks catastrophic. Dump the effective configuration rather than the file you think is loaded.
# nginx: the configuration actually in effect, with the rules that short-circuit a request
nginx -T 2>/dev/null | grep -nE '^\s*(return|rewrite|deny|allow)\b'
# IIS: the URL Rewrite rules for the site
appcmd list config /section:system.webServer/rewrite/rulesOn Apache, read the effective .htaccess chain for the path, not only the vhost.
DNS. Confirm that the name resolves to the host you are debugging. A stale record, a split-horizon difference or a change still propagating means the box you are watching is not the one answering.
Permissions and SELinux. The web server cannot read the document root or connect to the upstream socket. A 403 or a 502 with no application log line. On an enforcing system the audit log names the denial and the fix is a policy module, not a mode change.
ausearch -m avc -ts recent | audit2whyOnly now, the application. Worker pools, runtime, dependencies, the code. These are individually expensive to investigate and cannot be the cause of a response the application did not produce.
Two details that prolong the hunt
A 429 has two possible authors. An edge limiter answers it with the edge's own headers and typically a Retry-After; an application limiter answers it from inside the app, with the app's headers and after the app did some work. The time to first byte and the headers separate them, and the fix is in a different place for each.
An error page can outlive its cause. Edges and reverse proxies may cache a 5xx for a short time, so the origin recovers and the edge keeps serving the stored error until it expires. An Age header on the error response says it is a cached copy. Purge the path at the edge, or wait out the TTL, before concluding that the fix did not work.
Make each layer prove itself
Every step above pairs a layer with a way around it: resolving straight to the origin bypasses the edge, requesting a backend from the balancer host bypasses the balancer, requesting the application port on localhost bypasses the web server. The first hop where the response changes from the error to content is the layer that owns the error. The method is mechanical on purpose, which is what makes it fast at any hour.
Straight answers
What causes an instant 503 with no delay?+
Something answered before the application ran: an edge rule, a balancer with no healthy backend, a suspension or maintenance page, a rewrite rule, DNS pointing elsewhere, or a permissions or SELinux denial. No latency means look above the application.
How do I tell whether the CDN or my server returned the error?+
Read the Server and Via headers and the body, then repeat the request with curl --resolve pointed at the origin. If the origin serves content and the edge serves the error, the edge owns it.
Why does HAProxy return 503 when my application is running?+
Because its health check says otherwise. A check path or port that changed, or a backend that fails the check for a reason the stats page shows, leaves the balancer with no server to use. Fix the check, not the application.
How do I see rewrite rules that might be blocking a request?+
On nginx, nginx -T prints the configuration in effect; grep it for return, rewrite, deny and allow. On IIS, appcmd list config /section:system.webServer/rewrite/rules lists the site's rules. On Apache, read the .htaccess chain for the path.
The next 5xx that makes no sense, start at the top of the stack.
Or talk to an engineer who already does.
Talk to an engineerYou’ve reached the end.
Put the reading to work
What are
you running?
Tell us about the workload, traffic and recovery requirements. Start with the server builder, or talk through the details with an engineer.
Written by the engineers who build and run the infrastructure it describes.Talk to an engineer