Never reboot a mystery: what to capture before a restart

Where the reflex is right, and where it is not

Turning it off and on again works on a wedged web worker, a stuck queue consumer, a laptop. The state that was wrong is discarded and rebuilt from nothing, because nothing of value lived in it. Databases, message brokers, storage daemons and anything with a write-ahead log are the opposite case. The state is the product. A restart does not discard the bad state; it discards the information about how it got bad, and on some engines it adds damage of its own.

What a restart erases

The process table and its timings: which process has been running since when, which one is in uninterruptible sleep and on what. The owner of every open socket and file handle, which tells you who was holding the lock or the port. The kernel ring buffer, which on a default installation does not survive a reboot and is where an out-of-memory kill, a filesystem error or a failing disk announces itself. The memory contents of the service, which a core dump can preserve and a restart cannot. The shutdown and boot timeline that separates a crash from a failed start. And on a table engine without crash recovery, the unclean stop itself is how an index header gets damaged, so the restart that was meant to fix the service can be the event that corrupts the tables.

The evidence kit, in minutes

Everything below is read-only. Run it from a shell on the box, redirect the output to a timestamped file, and copy that file off the machine before anyone touches anything else.

$ uptime; last -x | head -20; journalctl --list-boots | tail -3
$ dmesg -T | tail -200
$ ps -eo pid,ppid,stat,etime,wchan:24,cmd --sort=-etime | head -60
$ ss -tnp state established | head -100
$ free -m; df -h; df -i
$ vmstat 1 5
$ top -b -n 1 | head -40

For a database, add the engine's own view of itself while it is still answering, and the tail of its error log whether it is or not.

mysql> SHOW FULL PROCESSLIST;
mysql> SHOW ENGINE INNODB STATUS\G
mysql> SHOW GLOBAL STATUS LIKE 'Threads_%';
$ tail -500 /var/log/mariadb/mariadb.log

The first block answers whether the box has already rebooted, whether the kernel killed something, what is stuck and on what, who holds which connection, and whether the machine is under memory or I/O pressure. The second answers what the database thinks is happening: a long transaction holding locks, a replication thread stopped, a deadlock, a table reported as crashed.

Crash, or failed start

The most useful distinction in the log is between a service that crashed and a service that shut down cleanly and then failed to come back. The first needs the trigger found before anything is restarted, because restarting into a live trigger reproduces the crash. The second is a startup or policy problem: a changed configuration, a full disk, a permission, a unit that was not enabled. The data is not at risk in the second case, and treating it as a crash wastes the time that would have brought the service back. The tail of the service log shows which one it is: a shutdown complete line followed by an abort at startup is the second case.

The reversible path

Once the trigger is known, the sequence is the same on every stateful system. Stop the trigger, so the fix cannot be undone by another crash mid-recovery. Take the free snapshot: stop writes, rename the data directory aside or snapshot the volume, and keep one copy off the box. From that point each step can be reversed. Repair before you restore, because a backup faithfully carries whatever the system carried when it was taken, and if the fault rode in with the data, the restore re-plants it while discarding everything written since. Verify at the boundary that failed: the query that aborted, the table that reported zero rows, the replication position, not merely that the port is open. Roll the planned tuning only after the patient is healthy; recovery and change are two different operations.

A zeroed row count after a hard reboot is usually a damaged index over intact rows. On engines without crash recovery, a repair rebuilds the index from the data; a restore throws the data away.

The cure, as distinct from the recovery

A repair is recovery. The cure is removing whatever causes unclean stops, and converting tables that cannot survive one to an engine that can. Both belong in the follow-up, not in the firefight.