How Does Accelerator Scheduling Affect Multi-User Home AI?

Accelerator scheduling affects multi-user home AI by deciding which requests start, share each iteration, keep memory state, or wait behind other work.

Several household users can send prompts with very different costs: a short light-control question, a long document, an image, a voice request, or an agent that generates many calls. The accelerator cannot infer household importance from arrival time alone. A scheduler must combine queue order, token budgets, batching, priorities, memory admission, and model residency while outputs remain unpredictable. The sections below explain why the same hardware can feel fair, fast, or unusable depending on how those choices are made.

FIFO Treats Arrival Time as the Only Priority

A first-in, first-out queue is simple, but a long prompt or response can delay many short requests that arrived later.

LLM requests have unequal token costs, so fairness based only on request count can give one user much more accelerator time than another.

FIFO is predictable at low load but produces head-of-line blocking when household workloads become heterogeneous.

Continuous Batching Shares Iterations Across Users

Iteration-level schedulers can add new sequences between decode steps and remove finished ones without rebuilding one fixed batch.

Orca’s iteration scheduling improves utilization while allowing several users to progress together.

Sharing does not guarantee equal speed. A scheduler still decides how many sequences enter, how often each one advances, and whether a new prefill interrupts active decodes.

Priority Policies Protect Latency-Sensitive Requests

Voice control and short interactive chat may deserve faster admission than background summaries, embeddings, or image generation.

Llumnix addresses latency priorities for heterogeneous LLM requests.

Priority must include aging or quotas so background tasks eventually run and one preferred user cannot starve the rest of the household.

-15% OFF

ZimaBoard 2 - Mini Home Server for Your Big Idea

Single board computer zimaboard2

$339.00 - $519.00

Buy Now

Memory Admission Can Block a Request Before Compute

A request needs KV cache and workspace in addition to model weights. The scheduler may delay admission even when compute units appear idle because memory headroom is insufficient.

ZimaSpace’s guide to multi-user memory pressure explains why longer contexts reduce how many conversations can remain active.

Preempting a sequence frees capacity but may require recomputing or restoring its state later, turning memory policy into additional latency.

Prefill and Decode Need Different Scheduling Treatment

Long prefills use compute heavily, while token decoding repeatedly reads weights and cache state. Running them together without control can stall streamed output.

Sarathi-Serve uses stall-free scheduling and chunked prefill to improve throughput while limiting latency disruption.

A home scheduler can reserve decode opportunities for active conversations and split large document prefills instead of letting one request monopolize a long iteration.

Fairness Must Be Measured in User-Visible Terms

Aggregate tokens per second can improve while one user waits much longer than another. Track queue time, time to first token, inter-token delay, completion time, and service share by user or workload class.

The Virtual Token Counter defines token-aware fairness rather than counting each request equally.

Use explicit classes for voice, interactive chat, agents, embeddings, and maintenance work. Then test overlapping long and short requests to confirm that the chosen policy matches household expectations.

Scheduling cannot create more accelerator capacity, but it can decide whether shortage appears as fair slowdown, tail-latency spikes, or one user blocking everyone else.