What Causes the Same Local LLM to Return Inconsistent JSON Schemas?
The same local LLM returns inconsistent schemas when its effective prompt or decoding constraints change, or when unconstrained sampling chooses different valid-looking structures.
A model file can stay identical while the server changes chat templates, system prompts, tool definitions, schema versions, sampling seeds, context truncation, or grammar support. Prompt-only JSON requests remain probabilistic, so optional keys, types, nesting, and extra prose can vary. Even constrained output can differ semantically when the schema permits several shapes or the runtime silently falls back.
Prompt and Context Differences Change the Requested Contract
Chat templates wrap messages with model-specific tokens, and tool frameworks may inject function descriptions or examples. Context trimming can remove the schema or a prior correction, while schema registry changes alter required and optional fields.
A production analysis of structured-output guarantees distinguishes prompt-only formatting, JSON mode, and grammar-constrained output. The signature is structural variation that follows template, context, or schema version rather than the model weights. This distinction remains visible during later household testing.
Log the exact serialized prompt and schema hash. Two UI requests that look identical are not controlled trials when hidden messages, tool order, or history differs. The intermediate result must remain inspectable before automation follows.
Free Sampling and Weak JSON Modes Do Not Enforce One Schema
Temperature, top-p, seed, and parallel execution influence token choices. Valid JSON mode can ensure braces and quoting while still allowing missing keys, alternative nesting, wrong types, or unexpected fields. That boundary should be measured separately under realistic operating conditions.
The schema-constrained generation benchmark evaluates constrained decoders across real schemas and separates coverage, efficiency, and output quality. Its design shows why parse success is weaker than exact schema compliance. The practical consequence appears when several sources compete for limited context.
If repeated runs all parse but validate against different shapes, the decoder is underconstrained or the schema permits variants. If outputs fail parsing, stop handling or truncation may be the earlier cause. This dependency should remain explicit in the final interface.
Runtime Fallbacks, Unsupported Keywords, and Repair Layers Alter Output
A local engine may not support every JSON Schema keyword, tokenizer, recursion pattern, or tool-call format. Some wrappers fall back to prompting, repair invalid text, or retry with another model without exposing the path. The result must therefore be checked against the original evidence.
The grammar-constrained decoding system compiles grammars to token masks for efficient structured generation. This mechanism depends on correct tokenizer and grammar state; unsupported coverage must be detected rather than assumed. This distinction remains visible during later household testing.
The failure boundary is varying field values within one valid schema. Structural consistency does not guarantee semantic correctness or deterministic content. Diagnose schema shape separately from whether the values are true. The intermediate result must remain inspectable before automation follows.
Freeze and Fingerprint the Entire JSON Generation Path
For every run, record model checksum, quantization, runtime version, chat template, serialized prompt hash, schema hash, supported-keyword report, grammar-cache key, sampling values, seed, context truncation, stop reason, validator result, repair attempt, fallback model, and final object shape.
Use local structured JSON as the expected capability boundary. Replay a schema matrix with fixed and varied seeds, then deliberately use an unsupported keyword and truncated context to verify that failures are explicit. That boundary should be measured separately under realistic operating conditions.
Require constrained decoding plus deterministic validation when schema shape is a contract. Reject silent fallback, version schemas, and keep semantic checks after parsing; a perfectly consistent object can still contain the wrong household action. The practical consequence appears when several sources compete for limited context.
Recommended products
Tech & AI HUB
More to Read
What Causes an AI Agent Planner to Repeat Steps It Already Completed?
Trace repeated planner steps through state persistence, completion evidence, tool-result parsing, context retention, retries, replanning, and stop conditions.
What Causes Permission Errors Only Inside AI Agent Subprocesses?
Compare parent and child identity, filesystem view, environment, capabilities, security policy, and executable path to diagnose subprocess-only denial.
What Causes CPU Saturation When Hardware Transcoding and Video AI Run Together?
Trace CPU saturation across codec offload, pixel conversion, frame copies, AI preprocessing, audio, subtitles, storage, and process scheduling.
