What we do · Operations
Edge traffic control for crawlers and scrapers
Some of the heaviest traffic your site receives is technically valid HTTP from search engines, AI crawlers, link previewers and scrapers. We control it at the load balancer, before it reaches PHP or the database, so the paths that cost real work stay fast for real visitors.
What it is
Rate limiting, classification and filtering at the edge for traffic that is not an attack in the classic sense, but behaves like one operationally:
- AI and search crawlers walking every page of a paginated sitemap at once.
- Link preview bots fetching the same uncached URL hundreds of times concurrently.
- Scrapers hammering faceted navigation, where every filter combination is a fresh database query.
- Crawlers that generate ever longer, re-encoded URLs on each pass.
These requests hit the uncached, expensive paths, and that is where we put the controls.
How we build it
Rate limits where the cost is. HAProxy stick tables track request rates per client and per agent on specific expensive paths, such as sitemaps, search and filtered listings. A crawler that exceeds the agreed rate is answered with a 429, and the rest of the site is unaffected.
Matching that survives change. Crawler identification uses substring rules that keep working when a bot changes the format of its user agent string.
Malformed patterns denied outright. Repeated double encoded URLs, impossible query strings and similar shapes are rejected at the edge with a single rule.
PHP protected from the requests that get through. PHP-FPM gets request termination timeouts and per worker memory ceilings, so one pathological request cannot run for minutes, grow without limit or take the worker pool down with it.
The cache keeps serving. Through all of it, Varnish keeps serving cached pages, so the public site stays fast while the uncached paths are protected.
Deployed everywhere, recorded once. Every guard goes onto every edge of the property, and you receive a short report with the traffic numbers, the rule and the effect.
What it protects you from
- An AI crawler or a link previewer firing hundreds of concurrent requests at your heaviest query.
- Scrapers taking a store's faceted navigation, and with it the database, out of action during a campaign.
- Runaway PHP workers growing until the operating system has to stop them.
- Protection rules that quietly stopped matching months ago.
- Blunt blocking that shuts out search engines and costs you rankings.
In practice
A standard part of every edge we build. On news and e-commerce properties the sitemap, search and filtered listing paths get rate guards from day one, sized to what the site can serve uncached.
Rules that follow the traffic, not a vendor list. We read your access logs to see which agents and request shapes cost the most PHP and database time, then write the rule for that pattern, deploy it on every edge of the property, and record the before and after numbers for you.
Why EuroVPS
- Controls at the load balancer, not a plugin inside the application.
- Rules targeted at the expensive paths, so search engines still crawl and visitors still browse.
- We read the access logs and measure, then write the rule.
- PHP-FPM limits and edge rules designed together.
- Guards rolled out to every edge of a property, with the numbers recorded.
FAQ
Will this hurt our search rankings? No. We rate limit expensive paths rather than blocking crawlers, and a 429 tells legitimate crawlers to come back later.
Can you block AI crawlers completely? Yes, if that is your policy. Many customers prefer to rate limit them instead.
Is this a WAF? It complements one. A WAF looks for attacks; edge traffic control manages load from traffic that is legitimate but expensive.
Do we need to change our application? No. The controls live at the load balancer and in the PHP-FPM configuration.
How quickly can you react to a new crawler? A targeted rule can usually be written and deployed across every edge within minutes of identifying the pattern.
Bots eating your capacity? Talk to an engineer.
Your next infrastructure partner
Let’s look at
your setup.
Tell us what you run today and what needs to improve. An engineer will review your brief and help you plan the next step.
+357 222 32280
who will look after your infrastructure.
Let’s start with your setup.
A little context helps us start the right conversation.