Your monitoring is a system too: test the path, not the dashboard
How to verify that logs actually arrive, that alarms reach a person, and that a quiet dashboard means a healthy network. A practical checklist with commands.
A quiet monitoring screen has two possible meanings: everything is fine, or the monitoring cannot see. From the screen alone you cannot tell which. Every monitoring stack is a chain of devices, agents, collectors, rules and notification channels, and each link can fail without raising any alarm of its own. This article is about proving the chain works, routinely, before the day you need it.
Silence is not evidence
Most monitoring designs carefully check the things being monitored and assume the monitoring itself. A switch is configured to send syslog, so its logs are assumed to arrive. An alarm rule exists, so a page is assumed to reach someone. A backup job is scheduled, so a configuration copy is assumed to exist. None of those assumptions is a measurement.
The places this chain typically breaks are mundane: a logging source interface or routing instance that cannot reach the collector, a firewall rule in between, a collector input that was reconfigured, a notification integration whose token expired, an alarm severity filter that quietly excludes the event type you care about. None of these makes a device look unhealthy from its own command line.
Check one: is every device actually being heard?
The device's view (show logging lists a server and says it is enabled) is a claim. The collector's view is the evidence. Turn it around and ask the collector: when did each device last send anything?
In Graylog, a search grouped by source over the last day shows every device that spoke; compare that list against your inventory, and anything missing is your finding. A simple end-to-end test from the device itself removes all doubt. On Cisco IOS and NX-OS, generate a harmless message and look for it at the collector within seconds:
send log 6 "monitoring path test from this device"
On a Linux host:
logger -p user.notice "monitoring path test from $(hostname)"
If the message does not appear, follow the path in order: source address and routing instance used for logging, reachability to the collector from that source, any firewall in between, then the collector's input and stream rules.
Do the same comparison for configuration backups. A device whose last saved configuration is weeks old has a broken backup path, even if nothing ever errored.
Check two: does an alarm reach a human?
Fire a real alarm through the whole chain on purpose: a test trigger, a dummy item crossing a threshold, or a log signature you inject. Confirm that the notification arrives in the channel your engineers actually read, with the right severity, within the time you expect. Do it after any change to the monitoring platform itself, and on a schedule even when nothing changed, because tokens expire and integrations drift.
In Zabbix, a test item on a monitored host plus a trigger at the severity you page on is enough. Push a value and watch it travel:
zabbix_sender -z <zabbix-server> -s "<host>" -k monitoring.selftest -o 1
Then confirm three things: the problem appeared, the notification was delivered, and the recovery was delivered when you reset the value.
Check three: are the severities honest?
Review which events reach a person and which do not. A short, explicit list of signatures that always page (routing neighbour down, link down, BFD down, VPN crypto failures, power, fan and temperature faults, storage network errors, firewall state tables near their limit) is easier to defend than a threshold someone set years ago.
A second opinion helps here. Rules filter the known noise first, then everything left is analysed in real time by AI models on our own NVIDIA A100 GPUs, inside our datacenter. The analysis reads an alarm alongside its recent history and related events and offers engineers a second opinion on what is likely going on. The alarm data never leaves our infrastructure, and the engineer still makes the call.
Check four: look from outside
Every view from inside your network shares your network's blind spots. A global measurement platform such as RIPE Atlas, with an anchor or probes measuring toward you, shows how real places on the internet reach you: latency, loss and path changes. When a carrier performs maintenance, the outside view tells you how long reconvergence took and whether anyone could have noticed, which no internal counter can.
For private circuits, a dedicated probe at each end measuring latency, jitter and throughput in both directions every minute, stored permanently, turns every conversation with a carrier from opinion into data. Exclude the probe traffic from the circuit's own statistics so the evidence stays clean.
Check five: sweep when nothing is wrong
Run read-only health sweeps on a schedule, not only after an alarm. Environmentals, routing and overlay state, redundancy consistency between pairs, optic levels and interface error counters change slowly, and slow change is exactly what threshold alarms miss. An optic drifting toward its low-light limit is a calm replacement this week or an outage next month.
Your monitoring self-test checklist
- Can you list every device that has not sent a log in the last day?
- Can you list every device whose saved configuration is older than its last change?
- When did you last fire a real alarm end to end, and did it arrive?
- Do the severities that page a person match a written list?
- Do you see your network from outside, continuously?
- Do you sweep when nothing is known to be wrong?
We monitor our network with Zabbix, Graylog, Observium and a RIPE Atlas anchor, and we run these checks against it on a schedule. If you want a provider that answers with measurements, or help testing your own chain, talk to us.
Published 22 October 2026.
All field notes