Chapter 3 — Grounded Answers and Citations

Course: Build and Evaluate RAG Applications

Prerequisites: You can read and run a short Python script. From Chapter 2 you need the idea that a retrieval pipeline cuts documents into chunks, each with an id, retrieves the top k chunks for a question with a score, and passes them to a model; the notes restate this in a sentence before they use it. Citation, claim, pointer, support, citation recall, citation precision and refusal are all defined in the notes before they are used.

What the reader can do after this chapter:

• Check, in code, whether each claim in a drafted answer points at a chunk that exists and contains the wording the claim relies on, and sort every claim into a named status.
• Compute citation recall and citation precision for a drafted answer, and say what each number leaves unchecked.
• Set a score threshold below which the system refuses to answer, and show with counts which questions that threshold wrongly turns away and which it wrongly lets through.

Core question this chapter answers: How do you make every claim in a generated answer traceable to a source chunk, and what should the system do when there is no chunk to point at?

Connections:

• Builds on: Chapter 2, which cut documents into chunks with ids and measured whether each question’s answer survived each stage. This chapter opens with that kind of chunk file and asks what the answer built from those chunks can prove.
• Leads into: Chapter 4, which studies why retrieval fails in production. It opens with the cases.tsv file your project writes here: each test question with the behaviour the system should show and the chunk that supports it.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question, so you can check that your answer has the right shape before you submit it.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines. Every line’s accepted value, or the rule that decides it, follows from what the task prints, so you can check your own file before you submit it.

Build and Evaluate RAG Applications — Chapter 3: Grounded Answers and Citations

Before you start

Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library. You do not need an account with any model provider or an API key: the project checks drafted answers that are printed in the project file and never calls a model.

Keep one question in mind while you read: when the assistant says something and shows a source, what exactly have I checked about that source? By the end of the chapter you should be able to name what a check proves, what it leaves open, and what the system should do when there is nothing to point at.

Summary

A grounded answer is one in which each claim can be traced to a passage in your documents. In a retrieval pipeline that means each claim carries a pointer, the id of a chunk, and the system can show the reader the text it points at. A pointer can fail in four separate ways: the claim carries no pointer, the pointer names a chunk that does not exist, the chunk exists but does not contain what the claim relies on, or the chunk contains something close that disagrees with the claim. Code can catch the first three cheaply and part of the fourth, and the share of claims and citations that pass is measurable as citation recall and citation precision. Grounding has a second half: when retrieval finds nothing that answers the question, the right output is a refusal that says so, and the threshold for refusing trades wrongly answering against wrongly turning people away.

Key concepts

Term	Definition
Grounded answer	An answer whose claims can each be traced to a passage in the documents the system was given, rather than to the model’s memory.
Claim	One statement in an answer that can be true or false and can be checked against a source, such as “a loan laptop must be returned within 14 days”.
Citation	A pointer attached to a claim that names the source passage it relies on. In this chapter the source passage is a chunk, and the pointer is its id.
Quote	A stretch of wording copied word for word from a chunk, carried beside a citation so that code can check it against the stored text.
Dangling citation	A citation that names a chunk id which does not exist in the store.
Unsupported claim	A claim whose cited chunk exists but does not contain the wording the claim relies on.
Citation recall	The share of an answer’s statements that are fully supported by their citations (Liu et al.’s definition).
Citation precision	The share of an answer’s citations that support the statement they are attached to (Liu et al.’s definition).
Refusal	An answer that says the documents do not contain what was asked, instead of answering from whatever text scored highest.
Score threshold	A cut-off on a chunk’s retrieval score: if the best chunk scores below it, the system refuses.
Retrieval score	A number a search assigns to a chunk for a question. In this chapter’s project it is the count of different non-stop words the question and the chunk share.
Chunk id	The name of a chunk, such as handbook-1, which makes a citation checkable.

Explanations

The failure, before the explanation

Here is a policy assistant that has been told to cite its sources. Its answers come back like this:

Question A: “How long can I keep a loan laptop?”

Reply: “A loan laptop can be kept for 30 days. [handbook-1]”

Question B: “Do I need a receipt for a 30 dollar claim?”

Reply: “Claims below 50 dollars do not need a receipt. [expenses-9]”

Question C: “Can visitors park overnight?”

Reply: “Visitors should ask reception for help. [network-3]”

Every reply carries a bracketed source, so all three look grounded. None of them is. In the first, the cited chunk exists, and it says “within 14 days of the issue date”, not 30. In the second, the sentence is correct and the chunk it names, expenses-9, does not exist: the reader who clicks it finds nothing. In the third, the documents say nothing about parking. The assistant picked the chunk that shared the most words with the question and made something of it, and the bracket makes that guess look checked.

An engineer used to compile errors expects a wrong reference to fail loudly. A citation fails quietly: the bracket is text, and it is there whether or not it is true. To see how to check it, you need to know what a citation is.

What a citation is: a pointer you can follow

Think of a footnote in a paper. The number in the text is worth something only if the note it leads to exists and says what the sentence says it does. A reference that leads nowhere, or leads to a different statement, is worse than none, because the reader trusts it. The bracket after each reply above, [handbook-1], [expenses-9], [network-3], is that kind of reference.

A citation is a pointer from a claim to a source, and it has to be one that a reader, or a program, can follow. In a retrieval pipeline the pointer is a chunk id.

A second way to see it: a commit message that says “fixes ticket 4412”. The reference is useful only if ticket 4412 exists and is about this change. A ticket id that exists but belongs to something else passes a glance and fails a check. Each of the replies above has the same shape, a reference present and unchecked.

That gives four ways for a claim to fail and one way to pass, and the project uses these names:

Status	What is wrong, if anything	How code can tell
UNCITED	The claim has no pointer	The list of cited ids is empty
DANGLING	A cited id is not in the store	Compare each cited id with the ids of the stored chunks
UNSUPPORTED	The chunk exists but does not contain the wording the claim relies on	Check that the claim’s quote appears, word for word, in a cited chunk
NUMBER_MISMATCH	The quote is there, but the claim states a number that the quote does not	Check that every number in the claim appears in the quote
SUPPORTED	Passed all four checks	None of the above fired

The last row is a statement about what code checked, not about whether the claim is true, a point the next section takes up.

A valid pointer is not the same as support

Return to the first reply, the 30-day claim with [handbook-1]. Its pointer is valid: the chunk handbook-1 exists, and the claim still disagrees with it. The same gap shows up in a provider’s tools. Sometimes a provider’s API does the pointer work for you. One provider’s citations feature splits documents into pieces to set “the minimum granularity of possible citations”: sentence chunking lets the model cite a single sentence or chain together several consecutive sentences, and for documents you supply as “custom content” the provider uses your content blocks “as-is and no further chunking is done” (Anthropic, Citations). That ties straight back to Chapter 2: if you hand over your own chunks, your chunks are the units a citation can point to, so how you chunked decides how precisely a claim can be traced.

The same page says what the feature promises: because the API “parses citations into the response formats described in the following sections and extracts cited_text directly, citations are guaranteed to contain valid pointers to the provided documents” (Anthropic, Citations). Read that sentence for what it says. It is a promise about the pointers: they lead to the documents you provided. It does not say anything about whether the claim the pointer is attached to agrees with the cited text.

A system that checks only that pointers resolve would pass the 30-day reply. So the checks go in layers: first, does the pointer lead anywhere; then, does the text it leads to contain what the claim relies on. A program can do the first reliably. For the second it can check the wording, and numbers in particular, but a quote that is present and a claim that goes beyond it, for example “checked weekly” added to a quote that says nothing about how often, passes the word-for-word test. That residue needs a person or an evaluation step, which later chapters build.

Measuring it: citation recall and citation precision

Take an answer with 5 statements and 8 citations. Four of the statements are fully supported by their citations, and six of the eight citations support the statement they are attached to. How well grounded is it? A count of brackets cannot say. Research on search engines that answer with citations gave the two measurements names. A trustworthy system “should cite comprehensively (high citation recall; all statements are fully supported by citations) and accurately (high citation precision; every cite supports its associated statement)” (Liu et al., Evaluating Verifiability in Generative Search Engines). The same study reported that, on average across the four engines it audited, “a mere 51.5% of generated sentences are fully supported by citations and only 74.5% of citations support their associated sentence”.

On the answer above, citation recall is 4 ÷ 5 = 0.80, because it counts statements. Citation precision is 6 ÷ 8 = 0.75, because it counts citations. An answer can have a high recall and a low precision (every statement supported, but with many extra, irrelevant citations attached), or the reverse (every citation sound, but a number of statements with none). The two move independently, and you need both.

A second way to see it is a test suite. Recall is “how much of the behaviour is covered by tests that pass”, counted over behaviours. Precision is “how many of the tests are testing something real”, counted over tests. Counting over different things is what makes them different numbers.

Making the model show its evidence: quote first, retract what cannot be quoted

Imagine asking the model, for each claim it writes, to paste the exact sentence it relied on. If the pasted sentence is not in the chunk, you know at once, with no judgement needed. Two published techniques work this way, because both make the model produce something that can be compared with the source. One provider’s guide on reducing hallucinations describes grounding “in direct quotes”: ask the model first to extract word-for-word quotes from the documents and, if there are none, to say “No relevant quotes found.”, then to base its analysis only on those quotes. It also describes a way to verify with citations: “Make Claude’s response auditable by having it cite quotes and sources for each of its claims. You can also have Claude verify each claim by finding a supporting quote after it generates a response. If it can’t find a quote, it must retract the claim” (Anthropic, Reduce hallucinations).

The design that follows is to make each claim carry a quote as well as an id, as the project’s drafted answers do. Code can then check three things that the model’s word cannot settle: that the id exists, that the quote appears word for word in that chunk, and that the numbers in the claim appear in the quote. A claim that fails is retracted or regenerated, and one that cannot be fixed is dropped. A model that is asked for a quote and invents one is caught by a string comparison.

Refusing: when there is nothing to point at

Go back to the third reply at the start. No chunk answers the question, so no citation could be right, and the bracket was always going to be decoration. The correct behaviour is a refusal: say that the documents do not cover it. The same provider’s guide puts the first step simply: “Explicitly give Claude permission to admit uncertainty”, and says this simple technique “can drastically reduce false information” (Anthropic, Reduce hallucinations). It is a published recommendation from that provider, not a guarantee about any particular system.

How does the system know there is nothing to point at? The cheap signal is the retrieval score of the best chunk. The project’s score counts the different non-stop words a question and a chunk share, and here is how it comes out for eight questions: six whose answers are in the documents, and two whose answers are not.

Question	Best chunk’s score	Answerable?
q1 loan laptop days	4	yes
q2 lost laptop	3	yes
q3 guest password	3	yes
q4 receipt for 30 dollars	3	yes
q5 project code	4	yes
q6 receipt age limit	2	yes
u1 visitors parking	1	no
u2 meeting room bookings	2	no

A score threshold says: answer only if the best score is at least the threshold. With a threshold of 1, all six answerable questions are answered, and so are both unanswerable ones: two wrong answers. With 2, u1 is refused and u2 is still answered, because it scores 2. With 3, both unanswerable questions are refused, and q6 is wrongly turned away because its best chunk scores 2. On these eight questions no threshold gets all eight right. Seven of eight is the best this signal can do here: at 2 the one wrong decision is u2, an answer where a refusal was due, and at 3 the one wrong decision is q6, a refusal where an answer was due.

The same trade appears in a vendor’s retrieval setting. One vendor documents a score_threshold between 0.0 and 1.0: “a higher score_threshold will limit the results to more relevant chunks, though it may exclude some potentially useful ones” (OpenAI, Retrieval). A cut-off on score is a dial, and there is a loss on both sides of it.

Because the score alone cannot separate every case, the refusal decision is best made with the support check as well: draft an answer, run the checks above, and refuse if no claim passes them. Each layer catches cases the other misses. Which setting is right depends on which error costs more: an answer given where a refusal was due, or a refusal given where an answer was due.

Three situations you will meet, and what they mean for your design

Before the table, three situations:

1. A customer clicks the source link in an assistant’s answer and lands on a “not found” page. The answer was right, and the customer stops trusting the source links.
2. An answer to a billing question quotes a page that says a different number from the one in the answer. Nobody notices until an invoice is disputed.
3. An internal assistant answers a question about a product the company does not sell, citing a page about a similar product, and the user acts on it.

What you see	What was checked, and what was not	What to do instead
A source link that leads nowhere (situation 1)	The id was never compared with the stored ids	Reject any answer whose cited id is not in the store
A real page with a different number (situation 2)	The pointer resolved; the claim was not compared with the text	Carry a quote with each claim, and check the quote and the numbers in code
A confident answer to an unanswerable question (situation 3)	Nothing: retrieval returned the closest chunk, so the answer looked cited	Set a threshold, add the support check, and refuse when neither passes

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

What are the four ways a citation can fail, and what makes each detectable?

It can be missing (the list of cited ids is empty), dangling (the id is not in the store), unsupported (the chunk exists but does not contain the claim’s quote), or carry a number the quote does not. The first three are checked by comparing against the stored ids and the chunk text; the number check by comparing the digits in the claim with those in the quote.

Why is a valid pointer not proof that a claim is supported?

A valid pointer only shows that the cited document exists. One provider’s guarantee covers valid pointers to the provided documents, and says nothing about whether the claim agrees with the cited text. A claim can point at a real chunk and still disagree with it.

What is citation recall?

The share of an answer’s statements that are fully supported by their citations. It counts statements, so an answer with 4 supported statements out of 5 has a citation recall of 0.80.

What is citation precision, and how does it differ from recall?

The share of citations that support the statement they are attached to. It counts citations, not statements, so it can be low when many extra, irrelevant citations are attached even though every statement is supported.

What did the audit of four generative search engines report on average?

That 51.5% of generated sentences were fully supported by citations and 74.5% of citations supported their sentence. These are numbers for those engines in that study, not for your system.

Why have the model produce a quote beside each citation?

Because a quote can be checked by code against the stored chunk text. An invented quote is caught by a string comparison, which a bare id or the model’s own assurance cannot provide.

What does one provider's guide say to do when the model cannot find a supporting quote?

Retract the claim: the guide says that if the model cannot find a quote for a claim, it must retract it. Dropping the claim is better than keeping one with nothing behind it.

Why are your chunks the units a citation can point to?

If you provide your own content blocks, one provider uses them as they are with no further chunking. So a citation can be no more precise than your chunks, and how you chunked in Chapter 2 decides it.

What does raising the score threshold trade?

It lets fewer unanswerable questions through, and it turns away more answerable questions whose best chunk scores low. In the project’s data, a threshold of 3 refuses both unanswerable questions and also refuses q6, which is answerable.

Why add a support check to a score threshold when refusing?

A score cannot separate every case: in the project, one unanswerable question scores the same as one answerable question. A draft answer whose claims fail the checks is a second signal, and the two catch different cases.

Model answer

The question: “Our assistant cites sources, but I don’t trust the citations. How would you make every claim traceable, and what should it do when there is no source?” This is the question a lead engineer asks before approving an assistant that talks to customers, and an interviewer for a role building with models asks to hear a verification habit. A strong answer hits five beats, in this order:

1. Pointer — each claim carries the id of a chunk, and a citation is a pointer that must lead somewhere.
2. Evidence — each claim also carries a word-for-word quote, so there is something to compare.
3. Checks — code checks that the id exists, that the quote is in the chunk, and that numbers in the claim appear in the quote; a failing claim is retracted.
4. Metrics — citation recall and citation precision are measured, and the answer says what they leave unchecked.
5. Refusal — when nothing supports an answer the system refuses, with a threshold set on purpose and a trade named.

“I’d make every claim carry two things: the id of the chunk it relies on, and a word-for-word quote from that chunk. The id alone is a pointer, and a pointer can lead nowhere or to a different statement, so I wouldn’t trust it by itself. Code checks that the id exists in the store, that the quote appears in that chunk, and that any number in the claim appears in the quote. A claim that fails is retracted or regenerated, not shown. I’d measure two numbers: citation recall, the share of statements fully supported by their citations, and citation precision, the share of citations that support the statement they are attached to. I’d also say what they leave open: a quote can be present while the claim says more than the quote does, so a person or an evaluation step still has to look at a sample. And when no chunk answers the question the system should refuse, not cite the closest one. I’d set a threshold on the best chunk’s retrieval score, measure which answerable questions it wrongly turns away and which unanswerable ones it lets through, and combine it with the support check, because a score alone cannot separate every case.”

Missing a beat is a failure state. Without pointer the answer does not say what a citation is. Without evidence nothing can be checked in code. Without checks citations are decoration. Without metrics “it cites well” is an impression. Without refusal the system answers questions it has no source for.

Why this matters

• Trust is lost at the first bad link. A source that leads nowhere, or to a different statement, teaches the reader to ignore all sources. The checks make that failure visible before the reader meets it.
• Review needs something to review. A reviewer, a customer or an auditor can check a claim only if it names the text it relies on. A quote and an id turn “the model said so” into a statement someone can verify.
• Refusal is a feature to specify. What the system does when it does not know has to be decided on purpose, with a threshold and a measured trade, because the default is to answer.
• Talking to customers. When a customer asks “where did it get that?”, an engineer who can show the chunk, the quote and the check that passed gives a better answer than one who can only say the model cited a source.

After this chapter

The project for this chapter is Check Every Claim and Refuse When You Should (project.md). You will check eight drafted claims against a set of chunks and sort them into the named statuses, compute citation precision and the share of claims supported, try five refusal thresholds on eight questions and see which decisions each gets wrong, and finish by writing cases.tsv: each test question with the behaviour the system should show and the chunk that supports it.

Chapter 4 opens with that file. It studies why retrieval fails in production, and uses your cases to show what each failure looks like.

Hands-on Project — Check Every Claim and Refuse When You Should

Objective

Check eight drafted claims against a set of stored chunks and sort each into a named status, compute two citation measures, try five refusal thresholds on eight questions, and write a file of test cases, so that you can say what a citation check proves, what it leaves open, and what a refusal threshold costs.

What this project needs: Python 3 and its standard library only (re). No task needs an account with a model provider, an API key, a vector database or any payment. The chunks, the drafted claims and the questions are printed below, and every check is a string comparison, so every result can be checked exactly. All three tasks are required.

Your evidence file. Keep one plain-text file, chapter3-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print. Where a value is a list, write its items comma-separated with no spaces, in the order given.

The data and the rules for all tasks

Copy this into your Python file. The nine chunks are the chunks a retrieval pipeline made from three short policy documents (the output of the previous chapter’s setting of 30 words with an overlap of 10, printed here so that you do not need that chapter’s file). Each claim in CLAIMS is (claim id, claim text, cited chunk ids, quote).

import re

CHUNKS = {
    "handbook-1": "Loan laptops are issued by the equipment desk. A loan laptop must be returned within 14 days of the issue date. Staff who need a laptop for longer must ask",
    "handbook-2": "date. Staff who need a laptop for longer must ask their manager to approve an extension. Extensions are recorded in the asset register. A laptop that is lost must be",
    "handbook-3": "the asset register. A laptop that is lost must be reported to the security team on the same day. The security team will then lock the device remotely.",
    "network-1": "Guest wifi is available in every meeting room. The guest network password changes on the first Monday of each month. Staff devices must use the corporate network instead of the",
    "network-2": "Staff devices must use the corporate network instead of the guest network. Printers are only reachable from the corporate network. Visitors who need a printer should ask reception, who will",
    "network-3": "Visitors who need a printer should ask reception, who will print for them. Remote staff reach the corporate network through the VPN.",
    "expenses-1": "Expense claims are submitted through the finance portal. Claims below 50 dollars do not need a receipt. Claims of 50 dollars or more must include a receipt image. Receipts older",
    "expenses-2": "dollars or more must include a receipt image. Receipts older than 60 days will be rejected by the portal. Travel claims must also name the project code. Approved claims are",
    "expenses-3": "claims must also name the project code. Approved claims are paid in the next weekly payment run.",
}

CLAIMS = [
    ("c1", "A loan laptop must be returned within 14 days of the issue date.", ["handbook-1"], "must be returned within 14 days of the issue date"),
    ("c2", "A loan laptop can be kept for 30 days.", ["handbook-1"], "must be returned within 14 days"),
    ("c3", "Lost laptops are locked remotely by the security team.", ["handbook-3"], "lock the device remotely"),
    ("c4", "Guest wifi is available in every meeting room.", [], ""),
    ("c5", "The guest network password changes weekly.", ["network-1"], "changes weekly"),
    ("c6", "Claims below 50 dollars do not need a receipt.", ["expenses-9"], "do not need a receipt"),
    ("c7", "Receipts older than 60 days are rejected.", ["expenses-1", "expenses-2"], "Receipts older than 60 days will be rejected"),
    ("c8", "Remote staff must use the VPN, which is checked weekly.", ["network-3"], "reach the corporate network through the VPN"),
]

QUESTIONS = [
    ("q1", "How many days can a loan laptop be kept before it must be returned?", "answer", "within 14 days of the issue date"),
    ("q2", "What happens to a lost laptop after it is reported?", "answer", "will then lock the device remotely"),
    ("q3", "When does the guest network password change?", "answer", "on the first Monday of each month"),
    ("q4", "Do I need a receipt for a 30 dollar expense claim?", "answer", "Claims below 50 dollars do not need a receipt"),
    ("q5", "Which claims must name the project code?", "answer", "Travel claims must also name the project code"),
    ("q6", "How old can a receipt be before the portal rejects it?", "answer", "Receipts older than 60 days will be rejected by the portal"),
    ("u1", "Can visitors park in the car park overnight?", "refuse", ""),
    ("u2", "How long are meeting room bookings kept before the portal cancels them?", "refuse", ""),
]

STOP = set("the a an of to is are in on for be by and or it that this who which what how do i can will must from at with as when".split())

def words(text):
    return re.findall(r"[a-z0-9]+(?:-[a-z0-9]+)*", text.lower())

def norm(text):
    # lower-cased words joined by single spaces: ignores case and punctuation
    return " ".join(words(text))

def score(question, chunk_text):
    # number of different non-stop words that the question and the chunk share
    return len((set(words(question)) - STOP) & (set(words(chunk_text)) - STOP))

The claim-status rule. For a claim, take the first of these that applies:

1. UNCITED — the list of cited ids is empty.
2. DANGLING — at least one cited id is not a key of CHUNKS.
3. UNSUPPORTED — norm(quote) does not appear inside norm(chunk text) for any cited chunk.
4. NUMBER_MISMATCH — some run of digits in the claim text (found with re.findall(r"\d+", text)) is not among the runs of digits in the quote.
5. SUPPORTED — none of the above applies.

Task 1 — Check every claim

Write check_claim(claim), which returns one of the five status names for a claim, following the claim-status rule, and print the status of all eight claims in order.

Then compute two numbers:

• the share supported: the number of claims with status SUPPORTED, divided by 8;
• the pair precision: count every pair of (claim, cited id), one pair for each id in each claim’s cited list. A pair is sound when the id is a key of CHUNKS and norm(quote) appears inside norm(chunk text) for that chunk. The pair precision is the number of sound pairs divided by the number of pairs.

Stop there: eight claims, nine chunks. You do not need a model, a vector database or a command-line interface.

Add to your evidence file:

t1_c1: <status of c1>
t1_c2: <status of c2>
t1_c3: <status of c3>
t1_c4: <status of c4>
t1_c5: <status of c5>
t1_c6: <status of c6>
t1_c7: <status of c7>
t1_c8: <status of c8>
t1_supported_share: <share supported, 3 decimal places>
t1_pair_precision: <pair precision, 3 decimal places>

Before you submit, read claim c8 and its quote. Write one sentence for yourself on what the check cannot see there; it is not scored.

Output: one Python file with a claim checker, and ten evidence lines. The last two measure the same eight claims in different ways.

Task 2 — Try five refusal thresholds

This task stands on its own: it needs the data and rules above, which include the scoring function.

Write best_score(question), which returns the highest score(question, chunk text) over the nine chunks. Then, for each threshold t from 1 to 5, apply the decision rule: answer when the best score is at least t, otherwise refuse. A decision is correct when it matches the question’s expected behaviour (the third entry of each tuple in QUESTIONS).

Print the best score of every question, and for each threshold the number of correct decisions out of 8. For threshold 3, print the ids of the questions with expected behaviour answer that were refused. For threshold 2, print the ids of the questions with expected behaviour refuse that were answered.

Add to your evidence file:

t2_best_scores: <id=score pairs for the eight questions in order, comma-separated, such as q1=9,q2=9>
t2_correct: <the five counts for thresholds 1 to 5, comma-separated>
t2_best_accuracy: <the largest of the five counts>
t2_wrongly_refused_t3: <ids of expected-answer questions refused at threshold 3, or none>
t2_wrongly_answered_t2: <ids of expected-refuse questions answered at threshold 2, or none>

Before you submit, look at whether more than one threshold reaches your best count and, if so, which different decision each one gets wrong. Write one sentence for yourself on what that means for choosing between them; it is not scored.

Output: a scoring function and a threshold table, and five evidence lines.

Task 3 — Write the test cases file

This task stands on its own: it needs the data above only.

Write cases.tsv, one line for each question in QUESTIONS, in order, with three fields separated by single tab characters: the question id; its expected behaviour (answer or refuse); and the id of the first chunk in CHUNKS whose text satisfies norm(gold) in norm(chunk text), where gold is the fourth entry of the question’s tuple. A question with expected behaviour refuse has no gold text, so its third field is the word none.

Add to your evidence file:

t3_rows: <number of lines in cases.tsv>
t3_answer_rows: <number of lines whose second field is answer>
t3_chunk_ids: <the third field of every line, in file order, comma-separated>

Output: a cases.tsv of eight lines, and three evidence lines.

What carries forward

Keep cases.tsv and chapter3-evidence.txt. The next chapter opens with cases.tsv: it uses your eight cases, six that should be answered from a known chunk and two that should be refused, to show what each way retrieval fails in production looks like.