K10 — NUMA Systems: Memory Nodes and Local Allocation

Key Insight: On multi-socket servers, “local memory” and “remote memory” access latency can differ by up to 3x — NUMA is the source of this difference.

The Concept of NUMA

NUMA (Non-Uniform Memory Access) means each CPU core has local memory. Local access is fast, remote access is slow.

2-socket example:
CPU 0 → Local: 60ns, Remote: 100ns
CPU 1 → Local: 60ns, Remote: 100ns

NUMA-Aware Allocation

Linux supports three NUMA policies:

1. First Touch (default): allocate on the accessing node
2. Interleaved: round-robin across nodes
3. Membind: force allocation to specific nodes

$ numactl --localalloc ./program
$ numactl --interleave=all ./program
$ numactl --membind=0 ./program

Summary

NUMA is critical for multi-socket performance. Use numastat to monitor cross-node access.