Local LLM Sampling: Why Decoding Settings Change Repetition and Stability
Decoding settings change local LLM behavior by reshaping which next tokens remain eligible and how strongly probability differences influence each selection.
A home model can answer one factual question consistently yet become repetitive during a long voice session or unstable during creative writing. The weights are unchanged; the decoder is choosing from their probability distribution differently at every step. Temperature, truncation, repetition penalties, seed, and context together determine whether output collapses into familiar loops or wanders into low-probability continuations.
Temperature Rescales Confidence Before a Token Is Chosen
At each step, the model assigns logits to possible tokens. Temperature rescales those logits before normalization: lower values sharpen differences and favor the leaders, while higher values flatten them and give weaker candidates more probability. Greedy decoding skips sampling and always takes the current maximum.
A technical explanation distinguishes temperature and token filters: temperature changes relative probabilities across the distribution, top-k keeps a fixed number of candidates, and top-p keeps the smallest set reaching a probability mass. These controls are related but not interchangeable.
Low randomness can stabilize formatting and factual phrasing, yet it can repeatedly choose the same locally attractive continuation. High randomness can escape that groove but also admit errors. Stability therefore means controlled variance for the task, not simply the lowest temperature.
Dynamic Filters Change the Candidate Set as Confidence Changes
Top-p uses cumulative probability, so its candidate count expands when the model is uncertain and contracts when one token dominates. Min-p instead sets a threshold relative to the most probable token, adapting the cutoff to confidence without targeting a fixed cumulative mass.
Research on min-p sampling reports better creative quality and diversity than top-p in tested high-temperature settings. The mechanism matters locally because small or quantized models can have sharper or noisier distributions than the defaults assumed by a hosted interface.
Filters operate before the random draw, while penalties modify scores based on prior tokens. Combining aggressive truncation with a strong repetition penalty can leave awkward alternatives, making the output look unstable even though each setting seemed conservative in isolation.
Where More Randomness Stops Improving Naturalness
Creative diversity and reasoning accuracy do not move together. A high temperature can produce varied stories but damage arithmetic, code, citation fidelity, and structured output. Conversely, greedy decoding may be appropriate for constrained extraction while sounding rigid in conversation.
Research on selective sampling finds that high-temperature choices can degrade reasoning at sensitive token positions, motivating selective rather than uniform randomness. One global setting cannot optimize every part of every task. That distinction changes the resulting household decision.
The failure boundary appears when the output violates the task constraint: invalid JSON, unsupported claims, broken code, or repeated loops. Sampling cannot repair a weak prompt, missing context, an unsuitable model, or corrupted conversation history; it only changes selection from available probabilities.
Build a Reproducible Sampling Test Grid
Choose three representative prompts: deterministic extraction, ordinary conversation, and open-ended generation. Hold the model file, quantization, prompt, context, and maximum tokens fixed. Run each across a small grid of temperature and one truncation method, using both a fixed seed and several random seeds.
Track invalid outputs, exact phrase repetition, unique-token ratio, task accuracy, and latency. This isolates decoding from the coherence problem described in long-context coherence, where long conversation context can independently reduce answer quality. This boundary remains visible during later evidence review.
Select separate presets by workload, not one universal value. Reject a preset if it improves variety while failing the task’s hard constraints, and rerun the grid after changing model, quantization, prompt template, or repetition penalty.
Recommended products
Tech & AI HUB
More to Read
Multilingual Embeddings: How One Vector Space Connects Household Documents Across Languages
See how aligned embeddings connect documents across languages, why retrieval quality varies, and how to test cross-language evidence coverage locally.
Agent Memory Conflicts: Why Recent Corrections Can Lose to Repeated Older Facts
Learn how duplicate old memories overpower corrections, where recency rules fail, and how to test supersession in a private agent memory store.
Private Search Reranking: How a Second Model Changes the Final Evidence Order
See why first-stage similarity and second-stage relevance disagree, when reranking helps private RAG, and how to evaluate reordered evidence.
