"The model hallucinated" isn't a diagnosis, it's a complaint. The real work starts when you turn that vague report into a reproducible root cause -- and most of the time the cause isn't in the model, it's in your pipeline.
Not every hallucination has the same root cause. Classify the incident before touching code:
- **Pure factual invention** -- the model asserts something false with no related context in the prompt. Usually a case of misapplied parametric knowledge. - **Contradiction of the provided context** -- the model had the correct information in the prompt (retrieved document, tool result) and still answered something different. This is the most serious, because it means the model is ignoring the grounding you gave it. - **Overgeneralization of a pattern** -- confidently answers something that "sounds" right for that type of question but doesn't apply to the specific case.
The second type almost always points to a pipeline problem (context engineering, not the model). The first and third usually need prompt or model adjustments.
Everything you do depends on being able to reproduce the failure. Log the `request_id` of every call from day one -- it's free and solves 80% of debugging friction later.
Without the exact prompt and `request_id` saved, "reproducing" the bug means guessing what was sent to the model three days ago. Save the full payload (system, messages, tools) associated with every response that reaches production -- not just the output.
The most common mistake: you assume the model had certain information because "it's in the database," but you never verify it actually made it into the prompt. Before blaming the model, print the exact prompt that was sent and check:
- Was the relevant document retrieved by your RAG step, or did retrieval silently fail? - Did the context arrive complete or was it truncated by a token limit? - Did the tool result the model needed arrive with `is_error: true`, and did the model improvise a response instead of reporting the error?
If the model confidently answered something that wasn't in its context, it didn't hallucinate out of caprice -- it probably filled a gap your pipeline left it.
On models that support `effort`, a low level can make the model respond quickly without internally verifying its own response against the provided context. If the use case is precision-sensitive (legal extraction, financial data, medical answers), raise the effort level before redesigning the whole prompt.
A pattern that measurably reduces hallucinations: explicitly instructing that saying "I don't have that information" is a valid and preferred response over inventing something plausible.
This single adjustment usually significantly reduces context-contradiction hallucinations -- but it requires your context to actually contain the information when you need it, or the model will say "I don't know" too often (a different, but more manageable, problem).
When you find a reproducible hallucination case, don't fix it ad hoc and move on -- turn it into a permanent test case. A set of 30-50 real (not synthetic) cases that you run every time you change the prompt or the model is the only real defense against silent regressions.
An underused technique: ask the model to audit its own response against the context, in a separate second call.
This second call is cheap (you can use a smaller model) and detects context contradictions much more reliably than eyeballing the output. It's, in essence, the same principle as the reflection pattern I cover later -- just applied as a debugging tool instead of as part of the production flow.
The real discipline here isn't a single technique, it's the order: reproduce, audit context, adjust effort, force admission of uncertainty, and only at the end touch the underlying prompt -- with a regression set that prevents the next deployment from breaking what you just fixed.
Carlos Montiel is an enterprise AI solutions architect. He implements LLMs, Agents, RAG, and orchestrators for companies across Guatemala and Latin America. Reach out for a consultation.
Contact Carlos Montiel