Chapter 8 — Hallucination Detection and Containment

Course: Build and Evaluate RAG Applications

Prerequisites: You can read and run a short Python script that handles strings and counts items in a list. You know that a language model produces text by choosing the next token, so its output can read fluently without being true. You know what an evaluation case and a grader are: a case pairs an input with a rule that decides pass or fail, and a grader applies that rule. The notes define every other term before using it.

What the reader can do after this chapter:

• Split a model’s answer into separate claims and check each claim against a supplied source by requiring a word-for-word quote, naming each claim as supported, unsupported or needing review.
• Flag a question whose repeated answers disagree, and say what that disagreement does and does not show.
• Write a containment rule in code that decides, from the status of each claim, whether an answer is delivered, trimmed, held for a person, or replaced by a refusal to answer.

Core question this chapter answers: How do you build a system so that a statement the supplied material does not support is caught, and cannot reach a user as though it were established?

Connections:

• Builds on: model-graded evaluation, where a second model scores output and is itself checked against people; this chapter applies the same discipline to one failure, unsupported claims.
• Leads into: a lesson in Building AI Agents, which moves the control point from the answer to the boundary: filtering what goes in and what comes out, and enforcing policy in code rather than in a prompt.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines. Every accepted value, or the rule that decides it, follows from what the task prints.

Build and Evaluate RAG Applications — Chapter 8: Hallucination Detection and Containment

Before you start

Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library. You do not need an account with any model provider or an API key: every model answer the project uses is printed in the project file.

Keep one question in mind while you read: if this answer contains one false sentence, what in my system would stop it before a user acts on it?

Summary

A hallucination is a statement from a model that is plausible but not true, or not supported by the material the model was given. One research paper argues that models produce them because training and evaluation procedures reward guessing over admitting uncertainty, so a system cannot assume the model will volunteer doubt. Prompting techniques such as allowing “I don’t know” and asking for quotes reduce the problem; the provider guidance cited here says they do not eliminate it. So a reliable system adds checks outside the model. It splits an answer into claims, requires each claim to point to a quote and verifies in code that the quote really is in the source, treats disagreement between repeated answers as a warning, and applies a written containment rule that removes or blocks claims that cannot be traced.

Key concepts

Term	Definition
Hallucination	Model output that is plausible but not true, or not supported by the material the model was given.
Source	The text supplied to the model for an answer, such as a manual page. A claim is judged against it.
Claim	One checkable statement inside an answer. An answer with three facts holds three claims.
Grounding	Tying each claim to something in the source, such as a word-for-word quote.
Supported claim	A claim whose quote appears in the source and is at least three words long, so this chapter’s check accepts it. It is a mechanical status: it shows the claim is traceable, not that the quote proves the claim.
Unsupported claim	A claim whose quote does not appear in the source, or that has no quote.
Needs-review claim	A claim whose quote appears in the source but is too short for code to tell whether it supports the claim, so a person checks it.
Normalisation	The fixed clean-up applied before comparing text: here, collapse runs of whitespace into single spaces and ignore letter case.
Consistency check	Asking the same question several times and comparing the answers; disagreement is a warning sign.
Critical claim	A claim tagged by the system’s designer as one the user will act on, so a wrong one is costly.
Abstain	Return “I do not have enough information” instead of an answer.
Containment	Design that stops an unsupported claim from reaching a user as established fact.

Explanations

The failure, before the explanation

A support assistant answers questions about a pump from its product manual. A user asks what the housing is made of. The assistant replies: “The housing is made of aluminium and the pump is certified for drinking water.” The manual page given to the assistant says the housing is stainless steel and says nothing about certification. Both sentences were fluent, confident and formatted like the correct answers before them.

Nothing crashed, and no check failed, because no check existed. A user who is installing the pump near a drinking supply now holds a claim nobody supported. The failure is the combination: a plausible statement, no trace back to the source, and no step between the model and the user.

What a hallucination is

One provider’s guidance describes the phenomenon as a model generating text “that is factually incorrect or inconsistent with the given context” (Anthropic, Reduce hallucinations, retrieved 2026-10-04). A research paper describes “plausible yet incorrect statements instead of admitting uncertainty” (Kalai et al., Why Language Models Hallucinate, arXiv 2509.04664, retrieved 2026-10-04).

Two examples. The first is the pump answer above: inconsistent with the supplied manual. The second is an imagined case: an answer includes a reference to a document that is not in the supplied material. Nothing in the material contradicts it, and it is still unsupported. A second way to picture it: a confident student who, asked a question they cannot answer, writes the most plausible-sounding sentence rather than leaving the line blank.

• Applied use: this chapter focuses on claims that a supplied source can settle. That is the case a system can test, because the source is in hand.
• Practical consequence: a fluent answer is not evidence. The only evidence is a trace from the claim to the source.
• Selection: facts the supplied source does not cover cannot be settled by this chapter’s checks; they need their own source or a person.

Why the model does not volunteer doubt

A picture: on a multiple-choice paper with no penalty for wrong answers, leaving a question blank can only cost marks, so guessing is the sensible strategy. A second picture: an employee who is only ever praised for having an answer will start having an answer for everything.

The paper’s argument behind those pictures, in its own words: “we argue that language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty”, and “language models are optimized to be good test-takers, and guessing when uncertain improves test performance” (Kalai et al., retrieved 2026-10-04). That is a claim about incentives in how models are trained and scored, made by the authors of that paper.

• Applied use: do not rely on the model to say “I am not sure”. Make abstaining an explicit, permitted outcome, and build the check around the output.
• Practical consequence: an answer’s confident tone carries no information about whether it is supported.

The cheap defences, and their limit

Take a spec-sheet question whose answer is not in the supplied document. With no instruction, the model may name a plausible material. Add one line to the prompt, “if the document does not say, answer I don’t know”, and it now has a permitted way out. Each defence below is a line like that, and costs little.

Provider guidance lists basic strategies (Anthropic, Reduce hallucinations, retrieved 2026-10-04):

• Allow the model to say “I don’t know”: “Explicitly give Claude permission to admit uncertainty.”
• Use direct quotes for grounding: “ask Claude to extract word-for-word quotes first before performing its task.”
• Verify with citations: “You can also have Claude verify each claim by finding a supporting quote after it generates a response. If it can’t find a quote, it must retract the claim.”
• Restrict knowledge to the documents: “Explicitly instruct Claude to only use information from provided documents and not its general knowledge.”

The same page adds the limit that matters for design: “while these techniques significantly reduce hallucinations, they don’t eliminate them entirely. Always validate critical information, especially for high-stakes decisions.”

An example of why the limit matters: a prompt that says “only use the manual” can still produce one sentence not in the manual, and the prompt cannot check its own result. A second way to see it: a sign on a door saying “authorised staff only” reduces who walks in, but the lock is what keeps the others out. The prompt is the sign. The rest of this chapter is the lock.

Detection one: check each claim against the source in code

Take the pump answer and split it into claims, each with a quote the model must supply:

Claim	Quote supplied	Quote in the source?
The housing is made of aluminium	“housing is made of aluminium”	no
The pump is certified for drinking water	“certified for drinking water”	no

A program can check the last column with no model at all: normalise both texts (collapse whitespace, ignore case) and test whether the quote is a substring of the source. A quote that is not there means the claim is unsupported.

A second example, the other way round. If the claim “The housing is stainless steel” arrives with the quote “made of stainless\nsteel” (extra spaces, a line break), normalisation turns it into “made of stainless steel”, which is in the source, so the claim is supported. Without normalisation the check would wrongly reject a correct claim.

There is a gap, and the chapter builds it into the rule. A quote can be genuinely present and still not support the claim. Suppose the claim is “the pump is safe to run without water” and the quote is “dry running”. Those two words are in the manual, in the sentence that says dry-running damage is not covered by the warranty. Code can see the quote exists; it cannot see that the claim contradicts the sentence it comes from. This chapter uses a simple mechanical guard: a verified quote of fewer than three words is marked needs review and goes to a person (or to a model grader that has been calibrated against people). The three-word figure is this chapter’s illustration, not a published standard.

• Applied use: require a quote per claim, verify it by normalised substring, and sort claims into supported, unsupported and needs-review.
• Exam or real-world signal: a question that asks how to catch an invented citation points to verifying the quote in code; one that says “the quote exists but the answer is still wrong” points to the need for review.
• Practical consequence: the check needs no model call, only a string comparison, so it can run on every answer, and its results are reasons a person can read.
• Limit: a longer quote can also appear in the source and still fail to support the claim, for example a long sentence that the claim twists. “Supported” here means traceable by this check. For a claim the user will act on, the cited guidance’s advice is to validate critical information, so a person or a calibrated grader should still look at critical claims that pass.
• Selection: use it when the source is supplied text. Do not use it to establish facts outside the source; and treat a pass as “traceable”, not as “true”.

Detection two: ask again and compare

Example: ask five times what the housing is made of and the answers are steel, aluminium, steel, brass, steel. Three of five agree. Ask five times for the rated flow and all five say 40. The first question is a candidate for a hallucination; the second is not suspicious.

The same page lists a second technique: “Run Claude through the same prompt multiple times and compare the outputs. Inconsistencies across outputs could indicate hallucinations.” (Anthropic, Reduce hallucinations, retrieved 2026-10-04.)

A second picture: five witnesses give a detective different colours for the car. Whatever the real colour is, you now know the account is shaky. Five witnesses who all agree could still all be wrong, so agreement does not prove an answer correct.

For a mechanical rule, normalise each answer (trim spaces, ignore case), count the most common answer, and flag the question if that count is below a chosen number. The project flags a question when the most common answer appears in fewer than 4 of 5 answers; the 4-of-5 figure is this chapter’s illustration.

• Applied use: run it on questions where the source does not make the answer verifiable, or as a cheap alarm before spending a person’s time.
• Practical consequence: it multiplies cost by the number of repeats, so use it selectively.
• Selection: it complements the quote check, it does not replace it, because it is silent when the model is consistently wrong.

Containment: what happens to a flagged claim

Detection is half the job. The other half is a written rule that decides what the user sees. Three situations first:

1. A pump answer has five supported claims and one unsupported claim about the colour of the casing. Nobody will act on the colour.
2. A different answer has five supported claims and one unsupported claim about the maximum safe pressure rating. The user will act on that figure.
3. An answer has one claim whose quote exists but is two words long.

The system’s designer tags claims the user will act on as critical. Here is a containment rule that this chapter uses, written so a program can apply it, in this order:

1. If any unsupported claim is critical, abstain (return that the information is not available).
2. Otherwise, if any claim needs review, hold for review by a person.
3. Otherwise, if any claim is unsupported, deliver the answer with those claims removed.
4. Otherwise, deliver the answer unchanged.

Rule 1 comes first because a person’s review is no use if a critical claim is already known to be untraceable. A second way to see it: a pharmacy that finds one unverifiable line on a prescription does not dispense the rest and mention the problem afterwards.

Situation	What it means	What the rule does
Unsupported casing colour (1)	A cosmetic claim could not be traced	Remove it, deliver the rest
Unsupported pressure rating (2)	A claim the user will act on could not be traced	Abstain
Two-word quote (3)	Traceable but too thin to trust unaided	Hold for review

The rule is the chapter’s own design. What the cited guidance supports is the principle behind it: an unsupported claim should be retracted (“If it can’t find a quote, it must retract the claim”), and critical information should always be validated.

• Practical consequence: the rule lives in code between the model and the user. A prompt cannot enforce it, because the prompt is part of what failed.
• Selection: removing claims is acceptable only when the remaining answer is still correct and not misleading without them; if removal would change the meaning, abstain instead.

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

What is a hallucination, in this chapter's terms?

Output that is plausible but not true, or not supported by the material the model was given. The chapter focuses on claims that a supplied source can settle.

Why can a system not rely on the model to say "I am not sure"?

One research paper argues that training and evaluation reward guessing over admitting uncertainty. So abstaining must be an explicit outcome, and checks must sit outside the model.

Name three basic strategies for reducing hallucinations.

Allow the model to say it does not know; ground the answer in word-for-word quotes from the source; verify each claim with a supporting quote and retract any claim with none; or restrict it to the supplied documents.

Do those strategies remove hallucinations?

No. The guidance cited says they significantly reduce them but do not eliminate them, and that critical information should always be validated.

How does code verify a claim's quote?

Normalise the quote and the source (collapse whitespace, ignore case) and test whether the quote is a substring of the source. If it is not, the claim is unsupported.

Why can a verified quote still not prove the claim?

A quote can exist in the source and still not support the claim, for example two words lifted from a sentence that says the opposite. Short quotes go to a person for review.

What does disagreement between five answers to one question tell you?

It is a warning that an answer may be a hallucination. Agreement does not prove correctness, because a model can be consistently wrong.

What is a critical claim?

A claim the designer tags as one the user will act on, so a wrong one is costly. An unsupported critical claim makes the system abstain.

State the containment rule in order.

Unsupported critical claim: abstain. Otherwise any claim needing review: hold for review. Otherwise any unsupported claim: remove those claims and deliver. Otherwise deliver unchanged.

Why does containment live in code and not in the prompt?

A prompt asks the model to behave; code between the model and the user enforces the rule on whatever the model actually returned.

Model answer

The question: “Your assistant answers from supplied documents. How do you stop it telling users things the documents do not say?” A strong answer hits five beats, in this order:

1. Permission — make “I don’t know” an allowed answer and restrict the model to the documents.
2. Grounding — require a word-for-word quote for each claim.
3. Verification — check every quote in code, and treat thin quotes as needing review.
4. Signals — repeat risky questions and treat disagreement as a warning.
5. Containment — a written rule in code: abstain on an unsupported critical claim, hold for review, trim, or deliver.

“First I’d tell the model it may say it does not have enough information and must use only the documents. Then I’d require a word-for-word quote for each claim, and check each quote in code after normalising whitespace and case; a quote that isn’t in the source makes the claim unsupported, and a very short quote goes to a person, because a quote can exist without supporting the claim. For risky questions I’d ask several times and flag disagreement, knowing agreement doesn’t prove it right. Finally a rule in code between the model and the user: if an unsupported claim is one the user will act on, abstain; otherwise hold for review, or trim the unsupported claims and deliver. Prompts reduce the problem but don’t eliminate it, so the final check can’t be a prompt.”

Missing a beat is a failure state. Without permission the model is pushed to guess. Without grounding there is nothing to verify. Without verification a quote is only a claim about a claim. Without signals the checks are blind to the cases the source does not cover. Without containment the detections change nothing a user sees.

Why this matters

• Customer-facing assistants. A single invented figure in a support answer becomes the company’s statement. A check between model and user is what makes the statement traceable.
• Document question-answering. The answer’s value is that it comes from the document; the quote check is how you show it did.
• Regulated or high-stakes uses. The guidance cited says to validate critical information; the abstain rule is where that validation is enforced.
• Incident review. When a wrong claim reaches a user, the record of its quote and its status shows where the check failed.

After this chapter

The project for this chapter is Contain an Unsupported Claim (project.md). You will verify quotes against a source and mark each claim, flag questions whose repeated answers disagree, and apply a containment rule to six answers.

A lesson in Building AI Agents moves the control point to the boundary of the system: what is filtered on the way in and on the way out, and why a rule enforced in code cannot be talked out of by a user the way a prompt can.

Hands-on Project — Contain an Unsupported Claim

Objective

Verify an answer’s quotes against a source and mark each claim, flag questions whose repeated answers disagree, and apply a containment rule to decide what a user sees, so that you hold a small checker that catches an unsupported claim before it is delivered.

What this project needs: Python 3 with its standard library only. No task needs an account with a model provider, an API key, an internet connection or any payment. Every model answer the tasks use is printed below. All three tasks are required, and each can be done on its own, in any order.

Your evidence file. Keep one plain-text file, chapter8-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print.

Task 1 — Verify quotes against a source

A support assistant was given this source text (all one paragraph, shown wrapped):

SOURCE = ("The Series 4 pump has a rated flow of 40 litres per minute. It must be installed with the "
          "inlet below the water line. The pump housing is made of stainless steel. The warranty "
          "covers parts and does not cover damage from dry running. Replace the inlet filter when the "
          "flow drops below 30 litres per minute.")

It produced ten claims, each with a quote. Each row is (claim id, claim, quote). Copy them as written; the extra spaces and the \n in a07’s quote are part of what the model returned.

CLAIMS = [
    ("a01", "The rated flow is 40 litres per minute.", "rated flow of 40 litres per minute"),
    ("a02", "The inlet must sit below the water line.", "inlet below the water line"),
    ("a03", "The housing is made of aluminium.", "housing is made of aluminium"),
    ("a04", "Damage from dry running is not covered.", "does not cover damage from dry running"),
    ("a05", "The filter should be replaced when flow drops below 30 litres per minute.",
            "Replace the inlet filter when the flow drops below 30 litres per minute"),
    ("a06", "The pump is certified for drinking water.", "certified for drinking water"),
    ("a07", "The housing is stainless steel.", "made of   stainless\nsteel"),
    ("a08", "The warranty covers parts.", "The warranty covers parts"),
    ("a09", "The pump needs a service check.", "must be serviced"),
    ("a10", "The pump is safe to run without water.", "dry running"),
]

1. Write norm(text): collapse every run of whitespace (spaces, tabs, line breaks) into one space, remove spaces from both ends, and convert to lower case. It does nothing else.
2. Write status(quote) returning one of three strings, tested in this order: unsupported if norm(quote) is not a substring of norm(SOURCE); otherwise review if norm(quote) has fewer than 3 words (words are separated by single spaces); otherwise supported.
3. Apply it to every claim. Print the count of supported claims, the supported rate (supported ÷ 10, 3 decimal places), the unsupported ids and the review ids, each in claim-id order.

Add to your evidence file:

t1_supported_count: <number of supported claims>
t1_supported_rate: <3 decimal places>
t1_unsupported_ids: <ids, claim-id order, comma-separated, no spaces>
t1_review_ids: <ids, claim-id order, comma-separated, no spaces>

Output: two short functions, and four evidence lines.

Task 2 — Flag inconsistent answers

Four questions were each asked five times. Each row is (question id, the five answers):

ANSWERS = {
    "q1": ["40", "40", "40", "40", "40"],
    "q2": ["30", " 30", "30", "40", "30"],
    "q3": ["steel", "aluminium", "Steel", "brass", "steel"],
    "q4": ["yes", "no", "yes", "no", "unknown"],
}

1. Normalise each answer: remove spaces from both ends and convert to lower case.
2. For each question, find the number of times its most common answer appears.
3. A question is flagged when that count is below 4.

Add to your evidence file:

t2_top_counts: <the four counts for q1,q2,q3,q4 in that order, comma-separated, no spaces>
t2_flagged_ids: <flagged question ids, question-id order, comma-separated, no spaces>

Output: a short script, and two evidence lines.

Task 3 — Apply a containment rule

Six answers were already checked. Each answer is a list of claims; each claim is (status, critical), where status is "supported", "unsupported" or "review" and critical says whether the designer tagged it as one the user will act on:

ANSWERS = [
    [("supported", False), ("supported", True)],
    [("supported", False), ("unsupported", False)],
    [("supported", True), ("unsupported", True)],
    [("review", False), ("supported", False)],
    [("unsupported", False), ("review", True), ("supported", False)],
    [("unsupported", True), ("review", False)],
]

Write decide(answer) that returns one string, applying these steps in order and stopping at the first that applies:

1. If any claim has status unsupported and is critical: abstain.
2. Otherwise, if any claim has status review: hold_for_review.
3. Otherwise, if any claim has status unsupported: deliver_trimmed.
4. Otherwise: deliver.

Apply it to the six answers in order.

Add to your evidence file:

t3_decisions: <the six results in order, comma-separated, no spaces>
t3_abstain_count: <how many of the six are abstain>

Output: a decide function, and two evidence lines.

What carries forward

Keep chapter8-evidence.txt and your decide function. A lesson in Building AI Agents moves the same kind of rule to the boundary of a system: it asks what to filter on the way in and on the way out, and which rules must be enforced in code rather than written in a prompt.