Skip to content

What Is AI Hallucination? A Plain-English Definition

An AI hallucination is output from a language model that reads as confident and fluent but is factually wrong or has no basis in any source. The US standards body NIST, in its Generative AI Profile (NIST AI 600-1), calls it "confabulation": confidently stated but erroneous or false content that may mislead users. Common examples are invented case law, a part number that does not exist, or a "quote" from a contract that the contract never contained.

Why do language models hallucinate?

A large language model (LLM) produces text that is statistically likely, not text that has been checked. When it lacks the facts, it still writes a fluent answer. On top of that, the way models are trained and scored tends to reward guessing over saying "I don't know".

That second point is the core argument of Kalai et al., "Why Language Models Hallucinate" (2025): most benchmarks give no credit for abstaining, so a model that guesses scores better than one that admits uncertainty. Two practical consequences follow:

  • Rare facts are the danger zone. The more specific a piece of information, such as an internal pricing rule or a serial number, the less likely the model learned it during training.
  • Confidence tells you nothing. A wrong answer is phrased just as assertively as a right one.

What actually reduces the risk?

Blanket warnings about AI do not help much. Three safeguards do, and they work best together: grounding the model in your own sources, requiring a citation for every claim, and running evals that measure behaviour over time. None of them is sufficient on its own.

  • Grounding: the model answers from documents supplied with the request rather than from what it absorbed in training. A common way to do this is retrieval-augmented generation (RAG), which looks up relevant passages before each answer.
  • Mandatory citations: every claim must point to a verbatim quote or a location in a source. Anthropic's documentation suggests explicitly allowing the model to say "I don't know", extracting word-for-word quotes first when working with long documents, and retracting any claim that has no supporting quote.
  • Evals: a fixed set of test questions with known answers, re-run whenever the prompt, the model or the data changes. This shows when a system starts guessing more often than before.

Process rules matter as much as technique. An AI agent that drafts quotations or handles support tickets should stop and hand over to a person, following the human-in-the-loop principle, when it finds no basis for an answer, rather than fill the gap. Anything with consequences, such as prices, deadlines or safety instructions, needs a review step before release.

What should you watch for?

Hallucinations can be reduced but not ruled out, a point Anthropic makes in the same documentation. The useful question is where a wrong statement could cause harm, and whether it will be caught there before anyone acts on it.

For any AI deployment, ask three things. Where does the information in the output come from? Can a person check that source in one click? And what does the system do when it finds nothing relevant? A system that openly reports "no supporting source found" is worth more in daily operations than one that always has an answer. More terms around AI agents are collected in the AI glossary.

Frequently asked questions

Can hallucinations be eliminated completely?

No. Model providers such as Anthropic state that mitigation techniques reduce hallucinations but do not remove them. Anything with real consequences still needs checking before it is acted on.

Is a hallucination a software bug?

Not in the usual sense. The model is doing what it was built to do, producing probable text, but the content happens to be wrong. The fix is a system that ties claims to sources and tests them, not a patch.

Why does it matter more for AI agents than for chatbots?

An agent acts on its own output, for example by closing a ticket or entering a value in a form. A wrong assumption turns into a wrong action that nobody has read.

Is confabulation the same thing?

Yes. NIST's generative AI profile uses the term confabulation and notes that it is colloquially known as hallucination.

Sources

  1. NIST AI 600-1: Artificial Intelligence Risk Management Framework – Generative Artificial Intelligence Profile (July 2024) nvlpubs.nist.gov
  2. Kalai, Nachum, Vempala, Zhang: Why Language Models Hallucinate (arXiv:2509.04664, September 2025) arxiv.org
  3. Anthropic Claude Docs: Reduce hallucinations platform.claude.com