Route Alarms So a Human Sees Them

Every monitoring platform can detect a problem. The hard part is the last mile: getting the right alarm in front of a person who can act on it, within seconds, without burying that person under alarms that do not matter. This is how we route alarms in Zabbix, and how you can check whether your own chain really works.

Detection is cheap, attention is expensive

Most monitoring setups do not fail because they miss a problem. They fail because nobody reads what they produce. Email alerts land in a folder that fills faster than anyone can empty it. Dashboards turn red and stay red for weeks, until red means nothing at all. The engineer who finally notices is usually reacting to a customer, not to the monitor.

Adding more checks does not fix that. What fixes it is a routing design that treats an engineer's attention as the scarcest resource you have, and spends it only on things that need a human now.

Decide what is allowed to reach a person

Start with severity, and be strict about it. Every Zabbix trigger carries a severity, from information up to disaster. We route only the top two, high and disaster, to the on-call engineer, and every trigger at that level has to pass one test: if this fires at night, does someone need to act before morning? If the honest answer is no, it is a warning. Warnings belong on dashboards and in the daily review, not in someone's pocket.

In Zabbix that decision lives in two places: the conditions on the action (trigger severity is greater than or equal to High) and the severity filter on the user's media. Set both, and make them agree. An action that matches everything, combined with a media filter that quietly drops half of it, is a configuration nobody can reason about at three in the morning.

Then suppress the echoes. When a switch goes down, every host behind it becomes unreachable too. Trigger dependencies tell Zabbix that the host alarms depend on the switch alarm, so the engineer receives one alarm that names the cause instead of forty that name the symptoms.

Deliver it where the engineer already is

Our alarms leave Zabbix through a webhook into a dedicated operations channel in Mattermost, the chat our engineers work in all day and carry on their phones at night. Chat beats email for alarms for practical reasons. It is where people already look. It shows who has seen the message. And the conversation about the fix happens in the thread under the alarm, so the thread becomes the record of what was done and when.

A good alarm message carries enough to act on without opening three other tools: the host, the trigger name, the severity, the current value, how long it has been in that state, and a link straight to the graph. Write the message template once, carefully, and every alarm you ever send benefits from it.

Make ownership visible

An alarm that three people saw and nobody took is as bad as one nobody saw. Acknowledgement closes that gap. When the on-call engineer acknowledges a problem in Zabbix, with a short note, everyone else can see it is owned, and the note is the first line of the record. If your team has a second line, an escalation step on the action (a further notification when a high-severity problem is still unacknowledged after a set number of minutes) turns "someone will probably look" into a rule.

Planned work deserves the opposite treatment: a team that is paged by its own maintenance learns to ignore pages, and that habit does not stay inside the maintenance window.

Prove it works

Prove the chain after every change

None of this counts until you have watched a real alarm travel the whole way. After every change to the monitoring platform we fire a genuine alarm and time it from trigger to message. Our network team has written up exactly how to run that test, and how to confirm every device is still heard, in Test the alarm path, not the dashboard. Keep the timings you record. If the number creeps up between tests, something in the chain is getting slower, and you learned it from a test instead of from an outage.

Tune per host, never silence blindly

Shared templates are where alarm fatigue is born. One database server legitimately runs hot, someone raises the CPU threshold in the template to quiet it, and every other host in the estate has just lost a useful alarm. We never edit a shared template to tune one machine. Thresholds that need to differ are set as host-level macro overrides, for example that host's own value of {$CPU.UTIL.CRIT}, so tuning one machine never changes the alarms on another.

The same rule applies to switching triggers off. A trigger that fires all the time is telling you something: either the threshold is wrong for that host, or the host has a real problem. Disabling it to make the board green removes the only witness. Fix the threshold on that host, or fix the host.

Watch the watcher

A monitor that has stopped is worse than none, because you trust its silence. Two checks cover most of the risk. Inside Zabbix, a nodata trigger on each agent, such as nodata(/<host>/agent.ping,5m)=1, catches a host that has stopped reporting, which is a different and quieter failure than a host reporting trouble. Outside Zabbix, something independent has to notice when the platform itself goes quiet: a heartbeat the server sends on a schedule to a separate system, which raises its own alarm when the heartbeat stops arriving.

We also treat the monitoring platform as production: enforced database retention, and an audit from the hardware up through the operating system, database, daemon and alert path.

A second opinion before the engineer reads it

Rules filter the known noise first, then every alarm that is left is analysed in real time by AI models running on our own NVIDIA A100 GPUs, inside our own datacenter. The analysis looks at the alarm alongside recent events, related hosts and known patterns, and places a short second opinion next to it: what else changed nearby, and what this pattern has meant before. It does not decide anything and it does not page anyone. It saves the engineer the first minutes of looking around. Because the models run on our own hardware, monitoring data never leaves our network and is never sent to a public AI service.

If you build something similar, keep the same boundary. The AI annotates the alarm. A person owns the decision.

Check your own chain

Which severities reach a person, and is that decision written down?
Can everyone see who owns an open alarm?
When did someone last fire a real test alarm and time it end to end?
Does a switch failure arrive as one alarm or as forty?
Has anyone tuned a threshold by editing a shared template?
How many triggers are disabled right now, and does anyone know why?
What tells you that the monitoring platform itself has stopped?

If you would like a second pair of eyes on how your alarms reach your people, we are glad to look. Talk to an engineer

An alarm is a message to a person, not a line in a log