Why a VM bigger than one CPU socket is often slower

When a virtual machine is slow, the reflex is to give it more cores. On a modern two-socket server there is a point where that reflex turns against you, and the bigger VM runs slower than the smaller one it replaced. The point is the size of one CPU socket, and the reason is called NUMA.

What NUMA is, in one paragraph

A server with two processors is really two computers sharing a bridge. Each socket has its own memory controllers and its own bank of memory attached directly to it. A core reading memory attached to its own socket gets it at local speed. A core reading memory attached to the other socket has to cross the interconnect between the sockets, and that access is slower. This layout is non-uniform memory access: every core can reach all memory, but not at the same cost. Each socket with its memory is a NUMA node.

Why crossing the boundary costs more than it looks

A remote memory access takes roughly 1.7 times as long as a local one. That alone is a large penalty for anything that touches memory often, which is nearly everything. It is also not the whole bill:

• Cache coherency. The two sockets must keep their caches consistent. When threads on both sides work on the same data, cache lines bounce across the interconnect, and every bounce is a stall.
• Interconnect saturation. The link between the sockets has finite bandwidth. A busy VM spread across both sockets competes for it with everything else on the host.
• Scheduling. A VM whose vCPUs cannot all run at once on one socket waits for the hypervisor to find room across both, and its threads land wherever there is space, often far from their data.

These effects compound rather than add. A workload that would scale nicely inside one socket can lose a large share of its throughput once it spreads across two, and the extra cores you added are paying for the privilege.

Read your host's topology

Before sizing anything, look at what the host actually has:

lscpu | grep -E 'Socket|Core|Thread|NUMA'
numactl --hardware

Typical output on a two-socket host:

Thread(s) per core:  2
Core(s) per socket:  18
Socket(s):           2
NUMA node(s):        2
NUMA node0 CPU(s):   0-17,36-53
NUMA node1 CPU(s):   18-35,54-71

The numbers that matter are cores per socket and the memory per node, which numactl --hardware prints along with a distance table. The distances show how much more a remote access costs than a local one on this exact machine.

Size against physical cores, not hyper-threads.

The two sizing rules

From the topology come two rules that are easy to state and worth never breaking:

1. A VM never gets more vCPUs than one socket has physical cores. Leave headroom for the hypervisor itself, so on an eighteen-core socket a sensible ceiling is a couple of cores below that.
2. A VM never gets more memory than one NUMA node serves. Otherwise part of its memory is remote by construction, whatever the scheduler does.

A VM that fits both rules can live entirely inside one node, and a good hypervisor will keep it there.

Check where a running VM actually lives

On a KVM host you can see how a guest's memory is spread across nodes:

numastat -c qemu-kvm
virsh vcpuinfo <vm-name>
virsh numatune <vm-name>

If numastat shows a large VM with substantial memory on both nodes, it is paying the remote access penalty on a share of its work. Inside a Linux guest, lscpu and numactl --hardware show the topology the hypervisor presents to it.

When one socket really is not enough

Some workloads genuinely need more than one socket can give. There are better answers than one giant VM:

• Scale out. Two VMs that each fit in one node, behind a load balancer or as members of a cluster, usually beat one VM spanning both.
• Fix the bottleneck first. A VM that seems to need more cores is often waiting on storage, locks or a single busy thread. Adding cores to a workload that cannot use them changes nothing except the NUMA penalty.
• If it must be big, make it NUMA aware. Present the guest a virtual topology that mirrors the host, two virtual nodes matching the two physical ones, and pin each virtual node to its physical node. An operating system and application that know about NUMA can then keep threads near their data. Enterprise KVM managers, OLVM among them, expose NUMA pinning for this.

Bigger is not linearly faster anyway

Even inside one socket, doubling vCPUs rarely doubles throughput. Parts of every workload are serial, locks get busier as threads multiply, caches are shared, and the hypervisor needs more effort to schedule a wide VM. Doubling is commonly worth a fraction of the doubling, and past the socket boundary it can go negative. The honest way to size is to run the real workload at two sizes and measure. The smaller VM wins more often than people expect.

How we do it

On our OLVM private clouds and every other platform we run, both rules are applied to every VM: never more vCPUs than one physical socket has cores, never more memory than one NUMA node serves, sized from the host's real topology. We right-size to what the workload needs instead of rounding up.

If your VMs were sized by guesswork, or simply grew one request at a time, we are glad to look at the topology and the numbers with you.

Talk to an engineer

Related: Private cloud on KVM with OLVM · Performance and capacity engineering · Dedicated cloud on Apache CloudStack, your spec