AI hallucination: how to spot it and engineer it out of a production system

An AI hallucination is a fluent answer with nothing underneath it

An AI hallucination is model output that sounds plausible and confident but has nothing behind it: not the documents the system retrieved, not the records it can read, not the facts. That is the whole definition, and the useful word in it is *unsupported* rather than *wrong*. A hallucinated answer can turn out to be correct by luck and it is still a hallucination, because the system had no grounds for it at the moment it spoke.

Three things it is not. It is not a bug in the sense of a broken code path, because nothing malfunctioned; the model did what it does, which is produce the most plausible continuation. It is not randomness, because dropping the temperature to zero does not make an ungrounded claim grounded. And it is not lying, because lying needs an intent to deceive and there is nothing in there to hold one. Mahmut Ozer's review in the Turkish Journal of Psychiatry argues that *confabulation* is the better word for the same reason.

That review also carries a split worth keeping, because it maps onto two different engineering jobs. An intrinsic hallucination contradicts the material the system was given, and you catch it mechanically by comparing the answer against the retrieved text. An extrinsic hallucination adds something that cannot be checked against that material at all, and catching it needs a source outside the conversation. Most production systems are decent at the first and blind to the second.

One example makes the shape concrete. In Why Language Models Hallucinate, the authors ask a 600B open-source model for a specific person's birthday and tell it to answer only if it knows. Three attempts, three different dates, none of them right. Nothing in those answers signals a guess, and that is the whole problem: the confident wrong answer and the confident right answer are formatted identically.

Models guess because guessing scores better

The willingness to guess is trained in, at two stages. During pretraining, facts that appear once in the data cannot be learned reliably, and the same paper works that out as a floor rather than a tendency: if a fifth of the birthdays in the corpus appear exactly once, expect a base model to get at least a fifth of birthday questions wrong. No prompt removes a statistical floor.

The second stage is the interesting one, because it is a scoring problem rather than a modeling problem. Most benchmarks grade answers right or wrong and give no credit for saying "I don't know". Under that rule, as the paper puts it, "abstaining is strictly sub-optimal" - an honest model that signals uncertainty loses to an identical model that always guesses. Every leaderboard built that way pays models to bluff, and the models learned.

The consequence for anyone shipping a system is short. The model arrives biased toward answering, that bias is not a defect you can file a bug against, and "only answer if you are certain" in the system prompt is a request rather than a control. Everything that actually holds sits outside the model.

Hallucination, algorithmic bias and a plain wrong answer are three different failures

These three get merged constantly, usually in the same sentence, and the merge is expensive because each has a different detection method and a different fix. Buying a groundedness guardrail for a bias problem does nothing. Retraining for a stale-document problem does less than nothing, since it costs money and leaves the stale document in place.

Algorithmic bias is a systematic skew in outputs across a group of cases: the system is consistently harsher, or more generous, or simply less accurate for one segment than another. Two things separate it from hallucination. It is reproducible, so you see it in aggregate rather than in any single answer, and it is often perfectly well sourced, because the skew sits in the data or in the process the data recorded. NIST's SP 1270 makes the point that the field keeps hunting for bias in the computational layer, in dataset representativeness and algorithmic fairness, while the human and institutional sources sit unexamined. A citation requirement does not touch any of that.

The third one is the most common in practice and the least discussed: a plain wrong answer, correctly retrieved from a source that is out of date, superseded, or wrong in the first place. The model behaved perfectly. Your corpus did not. Check this before anything else, because it is cheap to check and it explains a surprising share of the complaints that arrive labeled as hallucination.

A team that cannot tell these apart will spend a quarter on the wrong remedy and still have the original problem, which is why the distinction is worth the paragraph. When a bad answer lands, ask one question: could the system have known this? If yes, it is a retrieval or a data failure. If no, and it answered anyway, that is the hallucination.

The loud failures end up in court, the quiet ones just ship

Two cases are worth knowing, because both have court records attached rather than anecdotes. In Mata v. Avianca, decided in the Southern District of New York on 22 June 2023, two attorneys filed a brief citing six judicial decisions that did not exist. ChatGPT had produced them, complete with quotations and internal citations. Judge Castel imposed a $5,000 sanction and wrote the line that still frames the issue: "Technological advances are commonplace and there is nothing inherently improper about using a reliable artificial intelligence tool for assistance. But existing rules impose a gatekeeping role on attorneys to ensure the accuracy of their filings." The tool was not the problem. The missing check was.

The second is closer to what most companies run. In Moffatt v. Air Canada, decided by British Columbia's Civil Resolution Tribunal on 14 February 2024, the airline's website chatbot told a customer he could apply for a bereavement fare retroactively. He could not. Air Canada argued it was not responsible for information supplied by the bot; the tribunal's answer was that "while a chatbot has an interactive component, it is still just a part of Air Canada's website", and it ordered the airline to pay $812.02 in damages, interest and fees. The sum is trivial. The precedent is that your assistant's answers are your representations.

Neither is the shape to plan around, though, because the everyday version never generates a headline. The AI hallucination examples that actually cost companies money look like this:

A support assistant states a returns window that is in no policy document.

An order-status bot describes a shipment for an order it never looked up, because the lookup failed silently and the model filled in the plausible remainder.

A sales assistant quotes a product configuration that reads perfectly and does not exist in the catalog.

A contract summarizer adds a notice period the contract does not contain, because most contracts of that type have one.

Each of those is a single sentence inside an otherwise correct answer, which is why nobody notices until a customer acts on it.

Those four bars carry most of the argument. Open-ended recall from the weights is catastrophic. Hand the model the document and the worst score drops by most of the distance, but the floor is still not zero, even for the best system on a task where the answer is sitting right there in the context window.

You catch these with instrumentation, not with spot checks

Reading a sample of answers finds almost nothing, because these failures are fluent by construction. Measuring works, and four measurements cover most of the ground.

Groundedness, per claim. Split each answer into its factual claims and check whether each is supported by the text that was retrieved. That is the job the public leaderboards automate, and a cheap version is easy to run yourself: a second model, given only the retrieved chunk and one sentence, asked whether the chunk supports the sentence. Check the checker first, on a set where you know the answers, because a judge that agrees with everything is worse than no judge.

Empty and weak retrieval. The highest-yield log line in the whole system, and most teams do not have it. Record every request where retrieval returned nothing above your relevance threshold, and record what the assistant said anyway. That list is where the invented policy clauses live. If the numbers surprise you, the retrieval pipeline needs the work rather than the prompt.

A fixed eval set from real traffic. A hundred to two hundred real questions with known-correct answers, drawn from actual tickets rather than written by the team that built the system. Run it on every change to the prompt, the model, the chunking or the embeddings. That last one matters more than it sounds, since changing the embedding model rebuilds the whole index and quietly changes what gets retrieved for every existing question.

Disagreement between tools and prose. If the agent called a tool, you have the tool's output and the sentence the model wrote about it. Comparing the two is a mechanical check with no model in the loop, and it catches the failures where retrieval worked and the summary drifted.

Human review then goes where the instruments point, which is the tail rather than a random sample: low groundedness scores, empty retrieval, unusual question shapes, anything touching money or an entitlement. Reading fifty random answers a week mostly confirms that the easy questions are still easy.

Seven controls that stop AI hallucinations before they reach anyone

You cannot make the model unwilling to guess, so stop trying and build a system where a guess has nowhere to go. Every control below narrows the gap between what the system knows and what it is allowed to say. None of them require training a model. All of them are ordinary engineering, which is the good news and also why they get postponed.

Two of those get skipped most often, so a note from practice. On a 500,000-record knowledge base we validate every numerical claim against its source chunk before the answer is returned, and a claim with no retrievable source is flagged as unverified rather than smoothed into confident prose. That rule is unpopular internally for about a week, because the assistant starts saying "I don't have that" out loud, and then it becomes the reason people trust it.

The second is the difference between reading and remembering. An agent wired into live systems does not have to recall your order statuses, because it can go and look; on a real-time telemetry agent every answer is assembled from tool calls made at question time, and stale intermediate results are dropped from the history so the next question cannot reason from them. That is more work than pasting an export into a vector store, and it removes a whole category of invention, which is why connecting agents to your data is usually the first thing worth doing on this list.

One structural choice sits above all seven. A model following a fixed sequence of steps has fewer places to invent than one deciding its own path, so the agent-versus-workflow question is partly a hallucination question. If the process is the same every time, do not give the model room to be creative about it.

And the honest limit: none of this gets you to zero. Stanford's audit covered commercial legal research tools, retrieval systems built by companies with deep domain data, and still measured hallucination in 17% to 33% of answers, concluding that the providers' claims of elimination were overstated. Plan for containment and labeling, not for a solved problem.

The bill for skipping this is legal now, not only reputational

Air Canada already showed the shape of the liability: what your assistant says is a representation you made, and "the bot said it" is not a defense that survives contact with a tribunal. That reasoning does not depend on Canadian law in any interesting way, and consumer-protection regimes elsewhere land in similar places.

In Europe there is a second layer. The EU AI Act's transparency obligations have applied since 2 August 2026, and they do not ask you to be right - they ask you to disclose that a person is dealing with a machine and to mark generated content. We took it apart in what the AI Act actually requires from a system that is already running, together with which deadlines moved and which did not.

There is a quieter exposure that has nothing to do with the AI Act. When a model invents facts about a named person, those are inaccurate personal data, and GDPR gives that person a right to have them corrected. The privacy group noyb filed a complaint with the Austrian authority on 29 April 2024 over exactly this, after OpenAI could not correct a hallucinated date of birth and said it could only filter or block outputs about the person instead. Blocking is not correcting. If your assistant discusses customers, employees or suppliers by name, that gap is yours too.

Set against that, the controls in the previous section are cheap. Most of the cost is agreeing to a lower answer rate, and that call usually sits with someone who has never had to explain a fabricated policy clause to a customer. It is the same pattern behind most deployments that quietly fail after launch: the system was measured on how much it answered rather than on how much of that was true.

Start with the log line nobody writes

One thing to do this week. For seven days, log every request where retrieval returned nothing above your relevance threshold, together with the answer your system produced anyway. Then read the list. In systems that have never looked, it is usually longer than expected and concentrated in a handful of question types, and it reorders the roadmap on its own.

Then pick the smallest control from the table that addresses what you found. Usually that is the refusal path, because it is about a day of work and it turns the whole list from silent invention into visible gaps you can go and fill.

We build and audit this kind of thing for a living, so if you would rather someone else ran the first pass over your assistant's logs, that is what the free audit is for. The log line stands on its own either way.

FAQ