Rate limit the expensive path, not the crawler

Blocking bots is easy and usually wrong. Search engines bring you visitors, link previews bring you clicks, and even aggressive AI crawlers are often traffic you would rather slow down than shut out. The better approach is to find where crawlers actually cost you money, which is a handful of uncached, database-heavy paths, and control the rate there, at the load balancer. Here is how we do it.

Find out what is expensive before you write a rule

The cost lives on the paths that miss the cache and reach PHP and the database: sitemaps generated on the fly, search, faceted navigation where every filter combination is a fresh query, and anything with a query string the cache does not know.

Your access log already tells you which ones. If the log records the request time (nginx's $request_time, for example, as the last field), sum the time spent per path:

awk '{split($7, p, "?"); t[p[1]] += $NF; n[p[1]]++}
     END {for (u in t) printf "%10.1f s %8d req  %s\n", t[u], n[u], u}' access.log \
  | sort -rn | head -20

Then look at who is spending it, by user agent, on those paths:

awk -F'"' '$2 ~ /sitemap|search|filter=/ {print $6}' access.log \
  | sort | uniq -c | sort -rn | head -20

Two lists, one evening's work, and you know exactly where a rule will pay off.

Rate limit with HAProxy stick tables

HAProxy can count request rates per client and per user agent in memory, at line rate, and act on them. A minimal setup that protects only the expensive paths:

backend st_src
    stick-table type ipv6 size 200k expire 10m store http_req_rate(10s)

backend st_ua
    stick-table type string len 128 size 20k expire 10m store http_req_rate(10s)

frontend fe_web
    # bind lines and certificates as usual
    acl expensive path_beg /sitemap /search
    acl expensive url_sub -i filter=
    http-request track-sc0 src table st_src if expensive
    http-request track-sc1 req.fhdr(user-agent) table st_ua if expensive
    http-request deny deny_status 429 hdr Retry-After 60 if expensive { sc_http_req_rate(0) gt 20 }
    http-request deny deny_status 429 hdr Retry-After 60 if expensive { sc_http_req_rate(1) gt 60 }

Three design choices matter here. Only expensive paths are tracked, so nothing else on the site is touched. The response is a 429 with a Retry-After header, which well-behaved crawlers understand as "slow down and come back", not as an error to punish you for. And the second table counts per user agent, which catches a crawler spread across many addresses that would slip under a per-address limit. The thresholds above are placeholders: set yours from what the path can actually serve uncached.

Identify crawlers carefully

Match user agents loosely

Crawlers change the format of their user agent strings, adding version tokens, URLs or new product names. A rule that matches the whole string exactly stops working the day it changes. Substring matching on the stable part (hdr_sub(user-agent) -i <name>) keeps working.

Do not block the search engine by accident

Verify a crawler before treating it specially. The major search engines publish how: a reverse DNS lookup on the client address should return a hostname in their domain, and a forward lookup on that hostname should return the same address.

host <client-address>        # should return a name in the engine's published domain
host <name-returned-above>   # should return the same client address

Genuine crawlers then get rate limited on the expensive paths like everyone else, and still crawl the rest of the site freely. Impostors get no favours.

Reject the shapes that are never legitimate

Some request patterns have no legitimate use: URLs re-encoded over and over until they are thousands of characters long, impossible combinations of parameters, the same percent-encoding applied three times. A single rule at the edge rejects them before they cost anything:

http-request deny deny_status 400 if { url_sub -i %2525 }

Adjust the pattern to what your logs show. The point is that the load balancer is the cheapest place in the stack to say no.

Protect PHP from what gets through

No edge rule is perfect, so the application tier needs its own limits. In each PHP-FPM pool, a termination timeout stops one pathological request from holding a worker for minutes, and a memory ceiling stops it growing without bound:

request_terminate_timeout = 60s
php_admin_value[memory_limit] = 256M
pm.max_requests = 500

Set these together with the edge rules, so the two layers agree on what a reasonable request looks like. Meanwhile Varnish keeps serving cached pages, so readers and shoppers never feel any of this.

How we run it

On the news and e-commerce properties we operate, sitemap, search and filtered listing paths get rate guards from day one, sized to what each site can serve uncached. New rules are written from the access logs, deployed to every edge of the property, and recorded with before and after numbers for the customer. When a new crawler appears, a targeted rule can usually be in place across every edge within minutes of identifying the pattern.

Check your own edge

• Which five paths cost the most PHP and database time last week?
• Who requested them, by user agent?
• Do your limits apply only where the cost is, or to the whole site?
• Do you answer with 429 and Retry-After, or with a block?
• When did someone last confirm the rules still match live traffic?

If crawlers are eating your capacity, we can read your logs with you and write the rules. Talk to an engineer

Related: Edge traffic control for crawlers and scrapers · High-availability web tiers, N+1 at every layer · Web application performance audits