Conntrack: when health checks pass and traffic stops

The decision is made once

Connection tracking exists so that a firewall evaluates policy once per flow instead of once per packet. The first packet is matched against the rules, the forwarding and NAT result is written to the state table, and every later packet of that flow is a hash lookup. netfilter conntrack, the pf state table in OPNsense and the connection table in Cisco Firepower all work this way. The entry lives until the flow closes or its idle timeout expires, and for an established TCP connection that timeout is measured in days.

The consequence is easy to miss: a change to routing, to a NAT rule or to an interface affects the next flow, not the flows already in the table.

Two ways this drops traffic quietly

The stale path. A route changes, an interface moves, a NAT pool is edited. New connections take the new path. A long-lived stream keeps its original egress interface or NAT binding, so its packets leave the old way, towards a next hop that no longer forwards them or without the translation the far end now expects. The sender's TCP stack retransmits patiently on a connection it still considers established, the receiver sees nothing, and no process on either side has failed.

The full table. nf_conntrack_max is reached, usually because of a scanner, a NAT'd client with thousands of short flows, or long timeouts on a busy box. The kernel logs one line and starts dropping the first packet of new connections while existing flows continue. A health check that reuses an open connection keeps passing; users opening new ones get timeouts.

Neither shows up in process monitoring. The daemon is up, the listener is bound, the rule is present.

Why the health checks pass

A ping, a TCP connect probe and a firewall's path simulator each create a new flow or evaluate the policy fresh, so each receives the current, correct decision. The green reading is true about the wrong object. The question to put to the firewall is not what it would do with a new packet, but what it is doing with the flows it already holds.

# the live flows from one source, with their state and NAT binding
conntrack -L -s <source>

# how full the table is
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max

# the kernel's own report of a full table
journalctl -k | grep -i 'nf_conntrack: table full'

A flow that predates the last routing change and shows an egress or a translation that no current rule would produce is the entry to look at.

What to monitor

Arrival, not liveness. Rows per minute landing in the collector's database, messages per minute at the syslog receiver, a replication position that advances. A flat line is the alarm, whatever the process state says.

Counters against expectation. The byte counter on the rule that carries a stream grows at roughly the stream's rate. A rule that should be busy and is nearly flat means the packets are matching a different rule or not arriving at all.

State-table occupancy. Alert on nf_conntrack_count as a fraction of nf_conntrack_max well before it fills, and on the kernel's table-full message, which otherwise sits in the log unread.

Synthetic checks shaped like the real traffic. A probe that holds a long-lived connection and pushes a record through it at intervals, verified at the far end, tests the thing that breaks. A fresh ping tests the thing that does not.

After every routing or NAT change, list the flows older than the change and decide on purpose whether to flush them.

The fix is a scalpel

Delete the state for the affected source. The next packet creates a fresh entry with the current decision and the stream resumes.

conntrack -D -s <source>

A reboot or a full flush also works, and takes every other live connection down with it. For a full table, the fix is to raise nf_conntrack_max together with the hash size, shorten the established timeout where days are not needed, and find the client that is creating the flows.