When a SAN switch says 90% and means full: a Brocade ipadmd log loop

Short version: one of our embedded Brocade SAN switches reported its flash at 90 %. The real figure was 100 %, with about two days left before system daemons would no longer be able to write to disk. The cause was a single log file, written by the switch’s own management-IP daemon after an IPv6 DHCP client was switched off. We confirmed every server behind the switch had a working path on the second fabric, restarted it in a three-minute window, and checked every Fibre Channel switch in the data centre for the same pattern. Here is what we found, how we found it, and how you can check your own switches.

What we saw

The switch is a Brocade 16Gb SAN switch for HPE c-Class BladeSystem, running Fabric OS 8.2.x. Its built-in health monitor, MAPS, had marked it MARGINAL with the rule FLASH_USAGE >= 90 %. Two other things were unusual: its CPU baseline was about three times that of its identical neighbours, and it spiked to 100 % for a minute or two several times a day. Forwarding was unaffected, since that happens in the switch ASICs, so on the surface the switch looked healthy.

How we found the cause

Fabric OS gives the admin account a restricted shell: there is no df, no ls and no top. The tool that does work is supportshow, Brocade’s read-only diagnostic dump. It runs for two to three minutes, uses a fair amount of CPU and changes nothing. Its operating-system section includes the filesystem table:

Filesystem     1k-blocks    Used  Available  Use%  Mounted on
/dev/root         384648  382620          0  100%  /
/dev/hda3         199168   17912     170976    9%  /core_files
/dev/hda2         385168  226960     138328   62%  /mnt

So the root filesystem was full, not at 90 %. Core files and the secondary firmware partition were fine.

The dump itself pointed at the culprit. A normal supportshow from this model is a few megabytes. This one was over 160 MB, and almost all of it was a single section: cat /etc/ipadmd_log.txt, the log of ipadmd, the daemon that manages the switch’s management IP address. That one file was 158 MB and contained the same two lines more than two million times:

usage: kill [ -s signal | -p ] [ -a ] pid ...
       kill -l [ signal ]

That is the error kill prints when it is called without a process ID. Something was calling it with an empty argument every few seconds, and logging the failure each time.

Why it happened

The beginning of the same log tells the story. When the switch booted, DHCP for IPv6 (DHCPv6) was enabled on its management port and the daemon started an IPv6 DHCP client. Shortly afterwards the port was configured with a static address and DHCP was turned off. At that moment the log reads Disabling DHCPV6 in Ipadm, and the very next line is the first of the two million kill errors. The daemon was trying to stop the IPv6 DHCP client using a PID file that did not contain a process ID, and it retried, without end, every six seconds or so. That is roughly 1 MB of log a day. The extra process spawning also explains the higher CPU baseline.

The comparison that confirmed it: the identical switch on the other fabric, in the same enclosure, went through the same DHCP-to-static change on the same day but never had DHCPv6 enabled. Its whole ipadmd log was 34 lines, and its root filesystem sat at 62 %.

Why it matters

A full root filesystem on a fabric switch does not break forwarding, which is exactly what makes it dangerous. What breaks is everything that needs to write: committing a new zoning configuration (a fabric-wide zoning change needs every switch to store it), saving configuration changes, and logging. A switch in that state can still look healthy right up until the next zoning change fails. MAPS also understated the problem, because its rule triggers at 90 % and simply stays there.

The fix

The admin account cannot delete the file, but the daemon recreates its log every time it starts. We confirmed that before acting: on both switches, the log only held entries from the most recent boot, even where several boots had happened. With DHCPv6 now saved as off, a restart would also stop the loop from coming back.

Restarting a SAN switch means its fabric goes away for the servers behind it, so we checked the other side first:

• On the partner switch: every server port online, its uplinks to the core healthy, MAPS status HEALTHY.
• On the storage arrays (showhost -d on HPE 3PAR): every server behind the switch logged in through the second fabric, on both array controllers.
• No other diagnostic or change work running anywhere on either fabric.

Then a plain reboot from the serial console. Ports went down and came back in about three minutes, about one minute of which was the power-on self-test (fastboot skips it if you need a shorter window). Storage traffic ran over the second fabric throughout. Afterwards the root filesystem was back to 62 %, the ipadmd log was 31 lines with no errors, MAPS was HEALTHY, and every server had both fabrics again.

Checking the rest of the fleet

We then ran the same read-only check on every Fibre Channel switch in the data centre, one switch at a time. Only this one switch had the loop. Two older edge switches still have DHCPv6 enabled on their management ports. They show a much slower variant: the IPv6 DHCP client restarts every few minutes and adds about 70 KB of log a day, which would take years to fill them. Because switching DHCPv6 off on a running switch is exactly what started the loop above, we will change it on those two only together with an immediate restart, one fabric at a time.

How to check your own switches

1. Take any MAPS FLASH_USAGE alert seriously, and do not take the percentage at face value.
2. Run supportshow (read-only, a few minutes of CPU) and look for the df table and the size of the /etc/ipadmd_log.txt section. A healthy log is a few dozen lines. The dump contains binary bytes, so use grep -a.
3. Look for DHCPV6 is enabled on management interface near the top of that log, and for repeated usage: kill lines.
4. If you are moving a switch’s management port from DHCP to a static address, check whether DHCPv6 is on as well. If it is, plan to restart the switch straight after turning it off.
5. Before restarting any fabric switch, confirm from the storage side that every host has a live path on the other fabric.

Two redundant fabrics and a read-only diagnosis turned what could have been a surprise on the day of a zoning change into a planned three-minute task. If you run Brocade or HPE c-Class SAN switches and want a second pair of eyes on yours, EuroVPS customers can open a ticket at any time.