How Does Batch Size Affect First-Token and Per-Token Latency?

Larger batches often improve total token throughput, but they can increase first-token delay and per-token latency through queueing, heavier iterations, and memory pressure.

A local model at batch one may underuse its accelerator because each decode step moves model weights for very little work. Combining requests amortizes that movement and raises throughput. Yet requests may wait for admission, long prefills can block decodes, and every active sequence adds KV cache and token work, so the best batch depends on whether the target is throughput, responsiveness, or goodput within a latency objective.

Batching Amortizes Model Work Across Requests

A batched matrix operation processes tokens from multiple sequences together, improving hardware occupancy and spreading fixed launch or weight-transfer costs across more useful work. Gains are largest before compute, memory bandwidth, cache capacity, or scheduling overhead becomes saturated.

The Orca iteration-level scheduling introduces iteration-level scheduling so completed sequences can leave and new work can join between decoding iterations. This avoids forcing every request to wait for the longest member of a static batch.

Batch size can mean configured maximum, admitted sequences, or tokens processed in one iteration. Those values differ under continuous batching, so performance reports should state which quantity was controlled and observed. This distinction remains visible during later household testing.

First-Token Latency Includes Queueing and Prefill

Time to first token starts when a request arrives and includes scheduler waiting, prompt prefill, and the first decode step. A policy that waits to assemble a batch can increase queue time, while admitting a large prompt can occupy an iteration and delay existing interactive requests.

A current continuous batching comparison explains how continuous batching admits and removes requests dynamically and how aggressive scheduling trades throughput against TTFT. It clarifies why batch size alone cannot predict first-token responsiveness. The intermediate result must remain inspectable before automation follows.

Chunked prefill can interleave long prompts with decode work, but chunk size becomes another tradeoff. Small chunks protect interactive tokens yet add scheduling overhead; large chunks improve prefill efficiency while creating longer blocking intervals. That boundary should be measured separately under realistic operating conditions.

Per-Token Latency Depends on the Live Iteration

During decoding, one step generally produces one token per active sequence. More sequences enlarge the batch and can increase the duration of each iteration, even while aggregate tokens per second rises. Unequal contexts, cache pressure, preemption, and distributed synchronization add variance.

DistServe prefill-decode disaggregation separates prefill and decode because they have different computational characteristics and service objectives. The design illustrates why optimizing aggregate throughput can conflict with both TTFT and time per output token. The practical consequence appears when several sources compete for limited context.

The failure boundary is selecting the largest batch from an offline throughput chart for an interactive home assistant. If queueing or token gaps violate the user-facing objective, nominal throughput is not usable goodput regardless of accelerator utilization.

Featured

ZimaCube 2 Personal Cloud Home NAS

ZimaCube2

$799.00 - $2,499.00

Buy Now

Map the Batch Goodput Frontier

Replay short chats, long RAG prompts, and background jobs at controlled arrival rates. Sweep maximum sequences and batched-token limits while measuring TTFT percentiles, inter-token latency, end-to-end latency, prompt and output throughput, queue depth, cache use, preemption, and rejected requests.

Use the continuous-batching behavior in continuous batching behavior to distinguish configured limits from the live batch seen by each iteration. Keep prompt and output distributions fixed, and report the percentage of requests meeting both first-token and token-gap objectives.

Choose the setting that maximizes completed work inside those objectives rather than raw tokens per second. Isolate or deprioritize background batches when they reduce interactive goodput, and retest after model, quantization, context mix, or scheduler changes.