Local AI NUMA Locality: Why Memory Placement Changes Accelerator Feed Rate
NUMA locality changes accelerator feed rate because host preprocessing and transfers are faster when CPU threads, memory pages, and the device share a nearby path.
In a multi-socket home workstation, every CPU core can address all RAM, but access cost is not uniform. A GPU or other accelerator is usually attached through one socket’s PCIe root complex. If preprocessing runs on another node and buffers are allocated there, data may cross the socket interconnect before reaching the device, adding contention and variable latency.
NUMA Makes Host Memory Distance Observable
A NUMA system divides CPUs and memory into nodes with different access distances. Linux commonly allocates a page on the node local to the CPU that first faults it. Thread placement during model loading or input preparation can therefore determine where large buffers physically reside.
The Linux NUMA memory policies documentation describes task, VMA, shared, bind, preferred, and interleave policies. It also notes that policies mainly affect pages allocated after the policy is installed, which makes initialization order important. This distinction remains important under realistic household operating conditions.
For local inference, the critical path can include tokenization, image decoding, tensor preparation, pinned buffers, and device transfers. Remote placement adds a host-side bottleneck even when the accelerator itself reports unused compute capacity. The intermediate state should remain visible during later diagnosis and review.
PCIe Topology Links an Accelerator to Particular CPU Nodes
The shortest host-to-device path normally runs through the CPU socket whose root complex owns the accelerator. Binding the worker’s CPU threads and allocation policy to that neighborhood can improve bandwidth and reduce variance, especially when large inputs or frequent transfers keep the link busy.
NVIDIA’s CUDA NUMA guidance includes NUMA recommendations and warns that automatic balancing can degrade GPU applications in some cases. It recommends inspecting the topology and tuning policy for the actual node rather than assuming node numbers.
Placement is a graph problem, not a rule that node zero is fastest. The correct pairing depends on motherboard wiring, IOMMU configuration, other devices, and whether multiple workers compete for the same memory channels or PCIe links.
Binding Can Hurt When the Workload Uses More Than One Node
Strictly binding memory to one node can exhaust its bandwidth or capacity while other nodes sit idle. A pipeline may use a GPU near one socket but also a capture card, NVMe device, or second accelerator near another. One placement can optimize transfers while slowing preprocessing or storage.
The NVIDIA GPU affinity project maps processes to CPU cores associated with GPUs and notes that correct affinity can stabilize performance. Its multiple modes illustrate why unique, contiguous, socket, and NUMA scopes fit different multi-process workloads.
The failure boundary is a single-device benchmark generalized to the whole server. Do not bind blindly on integrated-memory systems, single-node machines, or pipelines spanning several devices; measure end-to-end latency, bandwidth, and contention under the intended concurrency.
Benchmark the Topology, Not Just the Accelerator
Map CPU nodes, memory capacity, PCIe devices, and accelerator locality. Run the same inference workload under default placement, CPU-only binding, memory-only binding, and matched CPU-plus-memory binding. Record host-to-device bandwidth, page placement, tokens per second, and p95 latency.
If model shards come from network storage as in network model storage, separate file-read time from page placement and device transfer. Warm the same data for each run, then repeat with the intended concurrent workers to expose memory-channel contention.
Adopt binding only if the matched topology improves repeatable end-to-end results without starving another service. If gains disappear after warm-up or reverse under concurrency, leave placement flexible or isolate only the transfer-critical threads and buffers.
Recommended products
Tech & AI HUB
More to Read
Private Search Score Calibration: How Raw Similarity Becomes a Usable Confidence Signal
Learn why cosine similarity is not confidence, how labeled queries calibrate scores, and how to monitor thresholds when a private corpus changes.
Model File Memory Mapping: How Shared Pages Reduce Duplicate RAM Use
Understand how mapped model pages are faulted and shared, why RSS can mislead, and which caches and buffers still consume RAM per process.
Private AI Audit Trails: How Event Logs Reconstruct Agent Decisions
Learn what an agent audit trail must capture, why ordinary logs are incomplete, and how to replay a private workflow without exposing raw data.
