What to check on a new server before it carries production

A new dedicated server usually arrives with an operating system, a login and an assumption: that everything you paid for is present, healthy and configured to be used. Most of the time that is nearly true. The gap between nearly and fully is where slow databases, idle network links and surprise outages come from. Here is the acceptance check we think every server deserves before it carries production, with the commands to run it yourself.

Why acceptance testing gets skipped

A server that boots looks finished. The operating system does not complain about a memory module it cannot see, a RAID cache running without its battery, or a bond that sends every packet down one link. Each of these is invisible in normal use and expensive later, and each takes minutes to check on day one. After go-live, the same checks need a maintenance window.

Memory: all of it, balanced, error free

Confirm the system sees every module you ordered, and that they sit in a balanced layout across the memory channels of each processor. An unbalanced layout runs, but slower than it should.

dmidecode -t memory | grep -E 'Size|Locator' | grep -v 'No Module'
free -h
ras-mc-ctl --errors

Corrected memory errors in the first days of a server's life are a reason to replace a module now, while it is easy, not a curiosity to watch.

Drives and the RAID controller

Read the health of every drive, including those hidden behind a hardware RAID controller, and the state of the controller's write cache:

smartctl -a /dev/sda
smartctl -a -d megaraid,0 /dev/sda    # a drive behind a MegaRAID-family controller
storcli64 /c0 show all                # Dell PERC and LSI family
ssacli ctrl all show config detail    # HPE Smart Array

Look for non-zero reallocated, pending or media error counters on any drive. A new drive with a corrected error is a drive whose own error correction is already working hard. On the controller, confirm the logical drive is the RAID level you ordered, the cache is in write-back mode and the battery or flash-backed cache module reports healthy. A controller that has dropped to write-through because of its battery can make an SSD array feel like spinning disk.

Then measure throughput on the empty volume before it holds data, with a tool such as fio, so you have a baseline to compare against later.

Is the flash being treated as flash?

Hardware RAID controllers often do not pass the solid state flag through to the kernel, so a volume built from SSDs is reported as a rotating disk and gets settings tuned for spinning media:

lsblk -d -o NAME,ROTA,SCHED,MODEL
cat /sys/block/sda/queue/read_ahead_kb

If ROTA says 1 on a volume you know is flash, confirm the media type from the controller first, then correct it with a udev rule so it survives reboots:

# /etc/udev/rules.d/60-ssd-tunings.rules
ACTION=="add|change", KERNEL=="sda", ATTR{queue/rotational}="0"
ACTION=="add|change", KERNEL=="sda", ATTR{queue/scheduler}="mq-deadline"
ACTION=="add|change", KERNEL=="sda", ATTR{queue/read_ahead_kb}="16"
ACTION=="add|change", KERNEL=="sda", ATTR{queue/nr_requests}="1024"

udevadm control --reload && udevadm trigger --action=change /sys/block/sda

Do not apply this to a volume on hard disks. Short read ahead suits flash and hurts sequential reads from spinning media.

Network: two links, both working

A server with two bonded ports is often running one of them. The Linux bonding driver's default transmit hash looks only at MAC addresses, and in a datacenter, where a host talks to a handful of gateway and peer MACs, that can put almost every flow on the same link.

cat /proc/net/bonding/bond0
ethtool -S <first port> | grep -E 'tx_packets|rx_packets'
ethtool -S <second port> | grep -E 'tx_packets|rx_packets'

Check that both members are up, that LACP has partnered with the switch, and that the hash policy is what you intended. Under load, both members' counters should grow at similar rates. For most servers, hashing on IP addresses and ports spreads traffic far better:

nmcli con mod bond0 bond.options "mode=802.3ad,miimon=100,lacp_rate=slow,xmit_hash_policy=layer3+4"

Match the switch side to it, so traffic is spread the same way in both directions. Then pull one cable, confirm the server stays reachable, plug it back, and repeat with the other.

Power, time and remote access

ipmitool sdr type "Power Supply"
chronyc tracking
getenforce

Both power supplies should be present and powered, and they should be fed from different power feeds. A server with two supplies plugged into the same feed has one point of failure with extra steps. Confirm the out of band controller (iLO, iDRAC or IPMI) is reachable on its management network and licensed for the remote console you will need. If SELinux is part of your standard, confirm it is enforcing before the first application is installed, not after.

Write it down

The last check is documentation: serial number, rack position, every port with its MAC address and what it connects to, every IP address, both power feeds and the cable IDs.

How we do it

Our dedicated servers are HPE ProLiant and Dell PowerEdge platforms, with out of band management on an isolated network. Every RAID build gets a drive health pass and a throughput test before it ships. Flash behind a RAID controller gets the tuning above, keyed to the media type read from the drives. Every bond hashes on addresses and ports, matched on the switch. We check the per-port counters to prove both links carry traffic.

If you have a server already in production that never had this check, it is not too late, and we are happy to go through it with you.

Talk to an engineer

Related: Bare metal servers on HPE and Dell · Performance and capacity engineering · Enterprise SAN storage on HPE 3PAR, all-flash