How Does Continuous Batching Affect Fairness on a Home AI Server?
Continuous batching improves utilization by refilling active batches, but fairness depends on how users receive admission, token service, and memory over time.
A home AI server can combine several conversations without waiting for every sequence to finish together. Completed requests leave, new requests enter, and active users share repeated inference iterations. That raises throughput, yet the requests are not equal: one user may submit a short command, another a long document, and another an agent that generates hundreds of tokens. A fair scheduler must decide what unit of work counts, how new requests enter, and how priorities interact with unpredictable output lengths.
Continuous Batching Changes the Scheduling Unit From a Fixed Batch to Iterations
Static batching keeps one group together until the group completes, wasting capacity when short requests finish early. Continuous batching can refill the open slots between generation iterations.
Orca introduced iteration-level scheduling so requests can join and leave as their sequence state changes.
This improves utilization, but it also means users repeatedly compete for a place in the next iteration rather than receiving one indivisible request slot.
Admission Order Determines Who Starts Accumulating Service
A request outside the active batch receives no model progress. The scheduler may admit by arrival time, estimated length, priority, available KV blocks, or a fairness counter.
vLLM combines continuous admission with paged KV cache management so memory can be allocated as sequences grow.
First-come, first-served is simple, but a queue of long requests can delay later short household commands even when those commands would finish quickly.
Counting Requests Equally Can Give Unequal Accelerator Service
A five-token answer and a five-hundred-token answer are both one request, yet they occupy very different numbers of decode iterations. Prompt lengths also create different amounts of prefill work.
The Virtual Token Counter defines token-based fairness because request count alone does not represent the service consumed by heterogeneous LLM workloads.
A household policy should decide whether fairness means equal token work, equal waiting time, equal completion opportunity, or priority for latency-sensitive tasks.
No single metric satisfies every workload. A voice command and a background summary should not necessarily receive identical treatment.
Unknown Output Length Makes Future Service Hard to Predict
The scheduler knows the prompt size at admission but usually does not know exactly how many output tokens the model will generate. One request can remain active far longer than expected.
Fairness research highlights unpredictable request lengths as a distinctive challenge for LLM serving.
Charging service as tokens are actually processed avoids relying entirely on a poor length estimate, but it may still allow a long request to occupy memory for many iterations.
Large Prefills Can Disrupt Users Already Receiving Tokens
A new document prompt may enter while several users are decoding. Its compute-heavy prefill can lengthen the iteration that active conversations must wait through.
Sarathi-Serve uses stall-free scheduling to split large prefills and reduce their effect on ongoing decode latency.
A scheduler that counts only decode tokens can still be unfair if one user repeatedly introduces large prefills that delay everyone’s streamed output.
Fair accounting should therefore include input processing as well as generated tokens.
Memory Pressure Can Create Fairness Problems Before Compute Saturates
Every active conversation needs KV cache, and longer contexts consume more blocks. A user with one large context can reduce the number of other requests that fit in the active batch.
ZimaSpace’s multi-user analysis connects household concurrency to shared model memory and scheduler decisions.
Preempting or swapping a request can restore capacity, but the interrupted user may later pay recomputation, cache reload, or a longer completion time.
Memory admission and compute scheduling must therefore follow the same fairness policy rather than operating as unrelated limits.
Priorities Need Aging, Quotas, and User-Visible Measurements
Voice control, accessibility tools, and short interactive chat may deserve higher priority than embeddings or overnight summaries. Pure priority scheduling can nevertheless starve low-priority work.
Llumnix uses dynamic scheduling to adapt request placement and resource decisions as serving conditions change.
Add aging, per-user quotas, maximum context or output limits, and a reserved background share so preferred work responds quickly without blocking everything else indefinitely.
Measure queue time, time to first token, inter-token delay, completion time, tokens served, and preemption by user or workload class. Continuous batching is fair only when the observed distribution matches the household policy, not merely when total tokens per second is high.
Recommended products
Tech & AI HUB
More to Read
What Features Enable a Home AI Trust Boundary Around Sensitive Files?
A home AI trust boundary combines encryption at rest, least-privilege permissions, runtime sandboxing, and scoped retrieval; no single feature holds alone.
What Causes Private Search Results to Favor Frequently Edited Files?
Frequently edited files gain ranking advantages when each update adds freshness, chunks, versions, or interaction signals without normalizing by source.
What Causes Smart Home Presence Models to Confuse Guests With Residents?
Guests can look like residents when the system observes household activity patterns but lacks a stable identity signal for the person producing them.
