Why Does Model Eviction Trigger Latency Spikes on Home AI Servers?

Model eviction creates a latency spike because the model that handled the previous request is no longer resident in fast memory. The next request must reload weights, restore runtime state, and process the prompt before normal token generation can begin. Once the model is warm again, later requests may feel fast.

If your home AI assistant responds quickly during an active conversation but pauses after sitting idle, switching models, or sharing the GPU with another service, the model itself may not be slow. The useful question is whether time is being spent loading the model or generating the answer. That distinction determines what to change.

Model Eviction Changes the First Request, Not Every Request

A local inference runtime keeps model weights in GPU memory, unified memory, or system RAM while the model is active. Eviction happens when the runtime removes some or all of that resident state. It may do so after an idle timeout, when another model needs the same memory, or when the service restarts.

A warm request can move directly into prompt processing because the weights are already available to the inference engine. A request after eviction follows a longer path: locate the model files, read the weights, place them in the required memory tier, initialize the execution path, and then evaluate the prompt.

This is why eviction usually appears as an isolated pause rather than a permanent reduction in tokens per second. The first response after a quiet period is slow, while the second request to the same model is normal. If every request remains slow, the bottleneck is more likely generation, CPU offload, memory bandwidth, queueing, or thermal limits.

AI Latency Is the Whole Chain, Not Just Generation Speed

Users often describe all waiting as “inference latency,” but a local AI request has several stages. It can wait in a queue, load a model, process the input prompt, and generate output tokens. A server can therefore report healthy generation speed and still feel unresponsive before the first token appears.

Model eviction mainly increases time to first token. It does not necessarily change the speed of the tokens that follow. That is why a tokens-per-second benchmark alone can miss the problem: the benchmark may start after loading has already finished or may reuse a model that has remained warm.

When the runtime exposes timing fields, compare them instead of relying on feel. Separate model-load and evaluation timings show whether the delay occurs before prompt processing or during it. A large load component on the first request and a small one on the next is strong evidence of a cold model rather than slow decoding.

Why Home AI Servers Evict Models

The simplest cause is an idle policy. A runtime releases inactive models so memory can return to the operating system or other applications. In Ollama, models remain loaded for five minutes by default, while keep-alive settings can extend residency. A pause that consistently follows the same idle interval points to policy rather than failing hardware.

Memory pressure creates a less predictable pattern. Two language models, an embedding model, an image generator, or a video service may all compete for RAM or VRAM. Systems sharing one server between Plex and local AI are especially vulnerable because a transcode or background job can displace a model even though the AI service itself has not been idle.

Model switching can cause the same churn. If only one large model fits comfortably, requesting Model B may force Model A out. Returning to Model A then triggers another load. The server appears randomly slow, but the spikes actually follow the order in which models are used.

Restarts are another boundary. A container update, service crash, host reboot, or manual stop clears resident state regardless of the keep-alive value. The first request after that event is a cold start by design. Treating it as an eviction bug can lead to aggressive settings that consume memory without improving normal operation.

Where the Eviction Delay Accumulates

The first cost is moving the model. Loading model weights into GPU memory typically requires reading them from storage into CPU memory before transferring them to the GPU. Larger files and slower storage paths extend that part of the wait.

The data path matters as much as the drive label. Weights may cross storage, system memory, and a PCIe or unified-memory path before inference starts. During that transfer, limited memory bandwidth can extend AI latency, especially when another workload is moving large amounts of data at the same time.

Loading the bytes is not always the end of the cold path. Depending on the runtime, the server may also create a GPU context, allocate memory pools, prepare kernels, or capture execution graphs. Systems that preserve initialized CUDA state can wake faster than a full restart because those setup steps do not all have to be repeated.

The prompt then needs to be evaluated again. Eviction normally discards the model’s active cache, so a long system prompt, retrieved context, or chat history must pass through prefill before the first new token appears. That delay is not weight loading, but users experience both costs as one silent pause.

What Different Latency Patterns Usually Mean

The timing pattern reveals more than a single speed test. Compare when the pause occurs, which metric grows, and what happens on the immediate repeat request.

The most characteristic eviction signature is the first row: one expensive request followed by normal responses from the same model. The other rows keep you from pinning a model in memory when the real issue sits elsewhere in the request path.

How to Test Whether Eviction Is the Cause

Begin with one model and one fixed prompt. Send the prompt twice with only a short gap, then repeat the test after the server has been idle long enough to cross its current unload policy. Keep prompt length and model settings unchanged so the comparison isolates residency.

Record total response time, model load time, prompt-evaluation time, and generation time where the runtime exposes them. If only load time expands after the idle period, the evidence points to eviction. If prompt evaluation grows instead, context length or cache reuse is the better lead.

Watch memory at the same time. A model should appear in RAM or VRAM after the first request and remain there during the warm test. If its footprint disappears before the slow request, you have direct confirmation that the runtime or another workload released it.

Finally, change one condition. Extend keep-alive, temporarily stop competing GPU services, or preload the model before the test request. A real eviction diagnosis should respond to the change. If latency remains unchanged, return to compute, memory, storage, or network bottlenecks instead of assuming the model was unloaded.

Practical Settings That Reduce Eviction Spikes

Keep the model that serves interactive requests resident for a period that matches actual use. A household assistant used every few minutes may benefit from a longer keep-alive window. A large model used once a day may not. The setting should protect the hot path, not turn every downloaded model into permanent memory consumption.

Leave real memory headroom. Installed VRAM is not identical to memory available for model weights because the display, runtime, context cache, and other applications also consume it. Checking used and remaining memory with nvidia-smi lets you measure real free VRAM before choosing model size and quantization.

Reduce the resident working set when the hot model barely fits. A smaller quantization, shorter context limit, or smaller default model can create enough margin to prevent routine eviction. Model and context choices should follow the workload’s RAM and accelerator requirements rather than parameter count alone.

Control model switching. Route common requests to one default model and reserve a larger specialist model for tasks that justify the reload. If several models must remain available, verify that their combined weights, caches, and runtime overhead fit instead of increasing the concurrency limit blindly.

Use fast local storage for model files and preload the interactive model after a planned restart. Faster storage cannot remove initialization or prompt-prefill work, but it can shorten the transfer stage. Preloading moves that cost to a controlled moment rather than making the first person who asks a question wait for it.

When Eviction Is Still the Right Trade-Off

Eviction is not automatically a fault. On a memory-constrained home server, it prevents an occasional AI workload from monopolizing resources needed for file sharing, containers, media services, or another model. The server gives up instant wake time in exchange for capacity and stability.

The right policy follows the workload. Keep a frequently used, latency-sensitive assistant warm. Allow batch models, image models, and rarely used experiments to unload. If two interactive models constantly displace each other, the durable choices are smaller models, more memory, or separate accelerators—not an infinite keep-alive value.

A practical boundary is repeat frequency. If users commonly return before the model would otherwise unload, extending residency removes visible friction at a modest memory cost. If requests are hours apart and the machine has other jobs, accepting one cold start may be the cleaner system design.

FAQ

Does model eviction make the AI’s answers worse?

Eviction changes readiness, not the stored model weights. Reloading the same model with the same prompt and settings should not reduce its capability. However, a discarded conversation or prefix cache can change how much context must be processed again, and missing application state can affect continuity if it was not stored separately.

Can a faster NVMe drive eliminate the latency spike?

It can shorten the weight-reading stage, especially when the previous storage path was slow or busy, but it cannot eliminate GPU setup, memory allocation, kernel preparation, or prompt prefill. If storage is only a small part of measured load time, an NVMe upgrade will not remove the whole pause.

Should every local model stay loaded?

No. Pinning every model can create the same memory pressure that caused the churn, while reducing capacity for caches and other services. Keep the small set of latency-sensitive models warm, let occasional models unload, and confirm the combined resident footprint under the server’s real workload.

The simplest rule is to compare the first request with the immediate repeat. A large load-time difference points to eviction; slow generation on both requests points elsewhere. Measure that boundary first, then tune residency, model size, and shared resources around the model users actually need to feel immediate.