Don't stop at the first fault.

Your site collapses. Someone finds a plausible fault and fixes it and declares victory. The site stays down, or comes back and re-collapses at the next trigger. The fix stopped at the first fault, and on the worst days there are four more behind it. This is the discipline that does not stop.

Rule one: rank by contribution, not loudness

The loudest symptom is a terrible sorting key. Example: the kernel screams disk I/O contention, but the engine of the cascade is a stalled resolver holding every worker, and the biggest ongoing driver is a session-file storm no first theory included. Rank causes by how much of the collapse they own, and fix in that order.

Rule two: read primary sources, not dashboards

Dashboards summarize. Kernels testify. When the layers disagree, the kernel wins. The canonical first pass on any wedged Linux box is the same short list every time, and then one heuristic worth framing: headroom present and the site down means an application or dependency fault, not capacity. A disk at fractions of a percent utilization while workers hang is lock contention, not I/O, no matter what the graph says.

# the canonical first pass on a wedged Linux box
dmesg -T | tail -50          # OOM kills, hung tasks, hardware complaints
journalctl --list-boots      # did this box reboot when you think it did?
last -x reboot | head        # corroborate the reboot story
cat /etc/resolv.conf         # who does this box think answers DNS?
iostat -x 5 3                # is the disk actually busy, or just blamed?
cat /proc/<pid>/stack        # where exactly is that worker stuck?
strace -p <pid> -c -f        # a few seconds: is it even making syscalls?

Ten minutes of primary sources beats an afternoon of dashboard archaeology.

Rule three: peel one fault at a time

One change, then re-measure at the boundary that failed, before touching the next layer. The discipline feels slow and is the fastest thing available, because two simultaneous changes that interact tell you nothing about either. Verification is per-layer: fix the resolver, prove name resolution at the box; clear the storm, prove worker states; then move up.

Rule four: distrust the we-fixed-it moment

When a fix holds and the symptom returns hours later, the reflex is to re-apply the fix harder. Re-measure instead. The question to ask is the simple one: how do you know it is doing I/O? A few seconds of strace that catch zero system calls mean the workers are blocked in the kernel on a lock, not doing I/O at all, and that is how the real driver gets found instead of papered over. When a symptom returns after a genuine fix, it is a different fault wearing the same clothes.

Rules five through seven: recovery has an order too

Rule eight: close the hole for everyone

Each new failure pattern becomes a named pre-flight check in the reference build. The method is not only how a bad morning gets untangled. It is how the same morning is kept from repeating anywhere else.