Change one thing, measure again: finding the real bottleneck
A repeatable method for slow servers: define the symptom, check each resource for use, saturation and errors, change one thing, measure again.
When a platform is slow, the quickest purchase order is for more of everything. It often works a little, costs a lot, and leaves the actual cause in place to bite again at the next traffic peak. The alternative is less exciting and far more effective: find the one resource the workload is waiting on, change one thing about it, and measure again. Here is the method and the tools.
Start with a number, not a feeling
"The site is slow" cannot be tested. "Checkout takes four seconds at the evening peak and under one at night" can. Before touching anything, pin the symptom to a measurement you can repeat: a page time from outside, a query duration, a batch job's run time, a request rate at which errors begin. That number is how you will know whether anything you change helped. Without it, every change feels like progress.
Then write down the path the slow operation takes: client, network, load balancer, web server, application, cache, database, storage. The bottleneck is somewhere on that path, and the path tells you where to look.
Check every resource for use, saturation and errors
A simple discipline, popularised by Brendan Gregg as the USE method, keeps the search honest. For every resource on the path, CPU, memory, network, storage, ask three questions: how busy is it, is work queueing for it, and is it reporting errors? A resource can be lightly used and still be the problem if requests queue in front of it, so the second question matters most.
CPU.
mpstat -P ALL 1
vmstat 1
pidstat -u 1
Look at per-core use, not just the average. One core at full load while the rest idle points at a single-threaded bottleneck that more cores will not fix. In vmstat, a run queue (r) persistently larger than the number of cores is saturation.
Memory.
free -h
vmstat 1
cat /proc/pressure/memory
Memory pressure shows as reclaim activity and stall time in the pressure file. On a well-built virtual machine there is no swap to hide in, so a workload that genuinely needs more memory than it has fails loudly, which is the honest outcome: the fix is more memory or a leaner application.
Storage.
iostat -x 1
pidstat -d 1
The columns that matter are the average wait times for reads and writes and the queue size. %util is misleading on flash and SAN volumes, which serve many requests in parallel and can show high utilisation while still having plenty of headroom. Rising wait times with a growing queue are the real signal.
Network.
sar -n DEV 1
sar -n EDEV 1
ss -s
cat /proc/net/bonding/bond0
ethtool -S <interface> | grep -iE 'drop|err|discard'
Compare traffic per member of a bond: one member busy while the other idles is a hashing problem, not a bandwidth problem. Drops and errors in the interface counters point below the operating system, at cables, optics or the switch.
Database. Many slow platforms are a database reading from disk what it should be reading from memory. On MySQL or MariaDB, compare logical reads with reads that had to go to storage:
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_read%';
If Innodb_buffer_pool_reads (reads that missed memory) grows quickly relative to Innodb_buffer_pool_read_requests, the working set does not fit in the buffer pool. The slow query log, with a sensible threshold, tells you which queries are paying for it.
Change one thing
Once one resource stands out, form a hypothesis that explains the original number, and make a single change that tests it: a buffer pool sized to the working set, a bond hash policy, a cache in front of the expensive page, a query given the index it needs, a VM resized to fit one CPU socket. One change at a time, because if you change three things and the number improves, you do not know which one helped, and you may have kept two that made things worse.
Measure again, the same way
Repeat the measurement from the start, under the same conditions: same time of day or same synthetic load, same test, same tools. A change that did not help is still a result, because it rules something out; revert it and go back to the resources.
Repeat until the original symptom is gone or the next bottleneck is something money really should solve. That is when buying hardware is the right answer, and you will know exactly which resource to buy.
What it looks like in practice
Some of the most effective settings are one line long. We size database memory buffers so the hot data lives in RAM and the flash serves real work, and every bonded server we build hashes on addresses and ports so both links carry traffic. Both are standing rules on every platform we manage.
How we do it
Performance work is how we operate every platform we manage, not a separate product. We measure at every layer, from per-volume I/O on the arrays and per-port counters on bonds and switches to slow logs on databases and outside-in timing through the CDN, before we name a cause. Standing rules that are reliably right, such as bonds that use both links, flash treated as flash and VMs that fit inside one socket, are applied to every build. When a platform does need more capacity, we tell you which resource, how much and why.
If your platform is slow and the only suggestion so far has been a bigger server, let us measure it first.
Published 22 October 2026.
All field notes