How Does NUMA Locality Affect Multi-GPU Home AI Inference?

NUMA locality affects inference when CPU threads, host memory, and GPUs communicate across nonlocal domains instead of staying near their PCIe path.

A dual-socket home AI server can expose two large RAM pools and several GPUs as one machine, yet access is not uniform. A worker scheduled on one socket may prepare tensors in memory attached to the other before transferring them to a GPU under a different PCIe root. The impact depends on model placement, host staging, tensor parallel traffic, interconnect topology, batching, and whether the workload is compute- or transfer-bound.

NUMA Turns One Memory Pool Into Distance-Dependent Access

In a NUMA system, each CPU socket or compute domain has memory that is closer to some cores than others. Software can address the combined capacity, but a remote access must cross an interconnect. That path usually has different latency and available bandwidth from local memory.

NUMA topology also affects GPU DMA because host pages may sit far from the GPU’s PCIe root complex. CPU scheduling and memory placement are separate decisions, and a virtual machine may not automatically see the host topology needed to align them.

The effect is small when host memory traffic is minor compared with GPU compute. It grows during model loading, CPU offload, tokenization, pinned-buffer copies, frequent synchronization, or workloads that spill beyond VRAM. NUMA capacity does not guarantee NUMA locality.

GPU Placement Adds a Second Topology to the Model Path

Multiple GPUs may attach to different CPU sockets, PCIe switches, or on-package partitions. A tensor moving between two accelerators can use a direct peer path, a dedicated GPU link, a PCIe switch, or a route involving host memory and an inter-socket hop. These paths are not equivalent.

Research on multi-partition GPUs finds that non-uniform access and inter-partition communication can amplify contention and kernel latency. Placement strategies differ for data shared globally, partially, or only within one workgroup or partition.

Model partitioning should follow the topology that carries the most repeated traffic. Adjacent layers or attention state placed across a slow boundary can communicate every token, while a less chatty split may tolerate distance. Counting GPUs without mapping their links hides the relevant relationship.

Tensor Parallelism Can Make Locality a Per-Token Cost

Tensor parallel inference divides operations within a layer across GPUs and combines partial results through collectives. That can fit larger models and use more compute, but communication repeats across many layers and tokens. A remote path therefore becomes a recurring cost rather than a one-time model-load penalty.

tensor parallelism works best when accelerator links and shard placement support the required synchronization. Adding a GPU across a weaker NUMA or PCIe boundary can increase capacity while delivering less throughput gain than the device count suggests.

Data parallel or request-level placement may be better when models fit independently and requests can remain local. Tensor parallelism becomes necessary when one model cannot fit on a single device, but its batch size, collective frequency, and interconnect determine whether the added capacity also improves speed.

Featured

ZimaCube 2 Personal Cloud Home NAS

ZimaCube2

$799.00 - $2,499.00

Buy Now

First-Touch and Thread Migration Can Break an Intended Layout

Operating systems often place memory near the thread that first touches each page. If initialization runs on one socket and inference workers later execute on another, the pages can remain remote. Scheduler migration can also move CPU preparation threads away from the memory and GPU they were intended to serve.

NUMA awareness connects local memory banks with the CPU sockets that access them most efficiently. Binding CPU threads without controlling memory allocation, or binding memory without aligning the GPU, solves only part of the path.

A stable layout may require CPU affinity, memory policy, device assignment, and topology-aware process launch. Containers and virtual machines add another mapping layer. The goal is not to bind everything blindly, but to keep high-volume producer, buffer, and consumer paths within the closest practical domain.

Locality Matters Most at Specific Inference Phases

Model loading emphasizes storage-to-host and host-to-GPU movement. Prefill processes many prompt tokens and can use larger matrix operations, while decode repeatedly advances one or a few tokens and may become sensitive to memory bandwidth, synchronization, and kernel launch overhead. NUMA effects can therefore change across one request.

GPU NUMA effects shows that placement-aware scheduling can improve attention by aligning work with memory domains and cache reuse. The lesson is narrower than a universal speedup: the gain appears where the kernel’s sharing pattern matches the topology-aware mapping.

A benchmark that reports only average tokens per second can hide slow first-token latency or poor scaling at a particular batch. Record load time, prefill throughput, inter-token latency, GPU link traffic, remote NUMA accesses, and CPU memory bandwidth separately.