Chapter 4 — Why Retrieval Fails in Production

Course: Build and Evaluate RAG Applications

Prerequisites: You can read and run a short Python script. From earlier chapters you need three ideas, each restated in a sentence in the notes before it is used: a retrieval pipeline cuts documents into chunks and returns the top k chunks for a question (Chapter 2), an answer can be checked against the chunk it cites (Chapter 3), and a set of known questions with expected answers can be kept to re-measure a system (Chapters 1 and 3). Stale index, re-ingest, retrieval miss, vocabulary mismatch, conflicting sources, source version, diagnosis ladder and regression are defined in the notes before they are used.

What the reader can do after this chapter:

• Run a diagnosis ladder on a failing question and name the first stage at which its answer was lost: missing from the documents, stale in the index, split across chunks, missed by the search, or in conflict with another source.
• Match each cause to the lever that repairs it, and apply a series of fixes while re-running every known case after each one to show which tickets it repaired and which it broke.
• Explain in a design review how to find out why a document assistant gave a wrong answer and fix it without making things worse, in five ordered steps.

Core question this chapter answers: When users report wrong answers from a retrieval system in production, how do you find the stage where each answer was lost, repair it, and show that the repair broke nothing else?

Connections:

• Builds on: Chapter 3, which checked that each claim in an answer points at a chunk that supports it. This chapter opens with the cases.tsv file your project wrote there, and asks what to do when an answer fails even though the citations look sound.
• Leads into: a lesson in Getting Started with AI, which asks whether a failing question should be repaired by changing the prompt, the information the model is given, or the model itself. It opens with the causes.tsv file your project writes here.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question, so you can check that your answer has the right shape before you submit it.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines. Every line’s accepted value, or the rule that decides it, follows from what the task prints, so you can check your own file before you submit it.

Build and Evaluate RAG Applications — Chapter 4: Why Retrieval Fails in Production

Before you start

Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library. You do not need an account with any model provider or an API key: the project diagnoses six failing questions from data printed in the project file and never calls a model.

Keep one question in mind while you read: a user says the answer was wrong; what is the cheapest check that tells me where it went wrong? By the end of the chapter you should be able to run those checks in a fixed order, name the first one that fails, and say what changing it might break.

Summary

A retrieval-augmented assistant that passes its tests can still fail in production, where documents change, questions are phrased differently from the documents, and more than one source holds the same fact. This chapter covers four of those failures: a stale index serving text from before a document changed, a chunk boundary that splits the answer, a search that misses the chunk holding the answer, and two retrieved sources that disagree while both look authoritative. Each leaves a different trace and has a different repair, so the useful habit is a diagnosis ladder: a fixed sequence of cheap checks, run in order, that names the first stage at which the answer was lost. A fix is also a change to the system, so each one is followed by a re-measurement of every known case, because a repair for one question can break another.

Key concepts

Term	Definition
Failure point	A place in a retrieval pipeline where an answer can be lost. One experience report on three real systems lists seven (Barnett et al.).
Stale index	An index whose chunks were built from an older version of a document than the one that is live now, so the search serves text that has since changed.
Re-ingest	Re-running the steps that cut, embed and store a document, so that the index matches the document’s current text.
Intact answer	An answer whose full wording sits inside one single chunk (Chapter 2).
Retrieval miss	A case where a chunk containing the answer exists in the index but is not among the top k chunks returned.
Vocabulary mismatch	A question and the passage that answers it use different words for the same thing, so a word-matching score does not link them.
Conflicting sources	Two or more retrieved chunks that state different values for the same thing.
Source version	A number stored with a document and with each chunk built from it, saying which edition of the document the chunk came from. It compares a chunk with its own document, not one document with another.
Edit date	The date a document was last changed. Unlike a version number that each document counts for itself, dates from different documents can be compared with one another.
Diagnosis ladder	A fixed order of checks, cheapest and most decisive first, run on a failing question to name the first stage at which its answer was lost.
Regression	A case that worked before a change and fails after it.
Known cases	A set of questions with their expected answers, kept to re-measure the system after every change (Chapter 1’s gold set, Chapter 3’s cases.tsv).
Lever	The kind of change that repairs a cause: add a document, re-ingest, re-chunk, raise k, or prefer the newest source.

Explanations

The failure, before the explanation

Here are four tickets that reached the support team of a policy assistant in one week.

Ticket 1: “The assistant says the guest wifi password changes every Monday. It changes on the first Monday of each month, and the handbook says so.”

Ticket 2: “I asked how many days I can keep a loan laptop. It said 14 and then, in the same answer, mentioned 30.”

Ticket 3: “I asked where staff connect from when working at home. It told me to check with IT. The page about the VPN exists.”

Ticket 4: “I asked whether a 30 dollar expense claim needs a receipt, and it said claims of 50 dollars or more need one, then stopped.”

An engineer who has worked with services will look for a crash, and there is none: every ticket is an answer that looks like an answer. Ticket 1 would pass any check on whether the answer is cited, because it cites a chunk that exists and contains exactly what it says. The chunk is out of date. Ticket 2 is two sources that disagree. Ticket 3 is an answer that was in the documents and never reached the model. Ticket 4 is Chapter 2’s split sentence again. Four tickets, four different causes, and nothing in the replies tells you which. To find out, you need the places an answer can be lost.

Where answers get lost: a list of failure points

Look at the four tickets again. Tickets 3 and 4 lost the answer on its way to the model; Tickets 1 and 2 lost it in the data the model was reading. A team that built retrieval systems in three domains, research, education and biomedical, wrote up what went wrong and listed “7 failure points to consider when designing a RAG system” (Barnett et al., Seven Failure Points When Engineering a Retrieval Augmented Generation System). Four of the seven are the stages this course has already built:

• Missing content (FP1): “a question that cannot be answered from the available documents”.
• Missed the top-ranked documents (FP2): “the answer to the question is in the document but did not rank highly enough to be returned to the user”, because “in practice the top K documents are returned”.
• Not in context (FP3): documents with the answer “were retrieved from the database but did not make it into the context for generating an answer”, which the paper ties to many documents being returned and then consolidated.
• Not extracted (FP4): “the answer is present in the context, but the large language model failed to extract out the correct answer”, which the paper says “typically” happens “when there is too much noise or contradicting information in the context”.

The other three concern format, specificity and completeness of the answer. Two things in the paper matter for how you work. Its abstract states two takeaways: “validation of a RAG system is only feasible during operation”, and “the robustness of a RAG system evolves rather than designed in at the start”. You should expect failures like the four tickets to appear after launch, and you should have a routine ready for them.

That routine is a diagnosis ladder. A second way to see it: a runbook for an outage. You do not start by reading source code; you check the cheapest, most decisive things first, in a fixed order (is the service up, is the network reachable, is the config current), and stop at the first one that fails. The ladder for a retrieval failure runs in this order:

1. Is the answer in the live documents at all? If not, it is missing content.
2. Is the index built from the current version of the document? If not, it is stale.
3. Does some chunk hold the whole answer? If not, the answer was split.
4. Is that chunk among the top k? If not, the search missed it.
5. Do the retrieved chunks agree? If not, the sources conflict.
6. If every check passes, the answer reached the model intact, and the fault is later.

The order is a design choice, and it follows cost and decisiveness: a text comparison against the live documents costs almost nothing and rules out a whole family of causes, while a retrieval re-run costs more.

Stale indexes: the right chunk, an old answer

Take Ticket 1. The live document says “The guest wifi password changes on the first Monday of each month.” The index holds a chunk saying “The guest wifi password changes every Monday.” It was built when the document said that, and nobody re-ingested after the edit. The assistant quoted the chunk it was given, cited it correctly, and was wrong. The citation checks from Chapter 3 pass this answer: the pointer resolves, the quote is in the chunk, the numbers agree. They compare the answer with the index, and the index is what drifted.

A second way to see it: a CDN that still serves last week’s page after a deploy. The cache is working as designed, and the content is what was true when it was filled. The remedy is the same family of ideas: know which version of the source each stored copy came from, compare it with the version that is live now, and refresh the copy that differs.

That gives the check. Store with every chunk the source version it was built from, such as a version number or an edit date. For a failing question, compare the version stored with the chunk against the live version of its document. A mismatch means stale, and the lever is to re-ingest: cut, embed and store the document again. This check costs a comparison of two numbers, which is why it sits second on the ladder, and it only works if the version was stored when the chunk was built. A chunk with no recorded version cannot be checked this way.

Chunk boundaries that split the answer

Ticket 4 shows what Chapter 2 taught. The sentence “Claims below 50 dollars do not need a receipt” was cut, with “Claims below 50 dollars” ending one chunk and “do not need a receipt” starting the next. The chunk that ranked highest began in the middle of the answer, so the model read a fragment.

The ladder’s check is the one from Chapter 2, “is the answer intact”: look for the expected wording inside a single chunk’s text. When no chunk holds it, the answer is split, and the lever is to re-chunk, with a larger size, an overlap, or sentence boundaries. A second example: a log line that wraps across two lines in a terminal. Grep for the whole message finds neither line. Joining the lines or widening the terminal fixes it, and neither changes the message. Chunk overlap plays the role of widening the terminal.

Retrieval misses: the answer is there, and the search does not find it

Ticket 3 is the clearest case. The user asks “Where do staff connect from home?”. The page that answers it says “Remote staff reach the corporate network through the VPN.” The question says connect and home; the page says reach and remote. A search that ranks by shared words sees only the word staff in common, and two other chunks about staff tie with it. With ties broken by chunk id and only two chunks returned, the chunk that answers is third and never returned. This is the second failure point in the paper: the answer is in the document and did not rank highly enough to be returned.

Two things combine here. The first is a vocabulary mismatch: the question and its answer use different words for the same thing, which Chapter 1 showed is the case embeddings are good at and word-matching is not. The second is the cut-off k. The cut-off is the lever the ladder offers first: raise k so the chunk makes the list. It is also the cheapest lever and the one with side effects, as the next sections show. A second example from outside search: a recruiter shortlisting from a CV pile by keyword. A candidate who writes “built deployment pipelines” is missed by a screen for “CI/CD” even though the experience is there. Widening the shortlist or changing what is matched on are the two repairs, and each has a cost.

Conflicting sources: two authorities, one question

Ticket 2 retrieved two chunks that both look official. One handbook chunk says a loan laptop must be returned “within 14 days of the issue date”. An older FAQ chunk says “within 30 days of the issue date”. Neither is a typo and neither is stale in the sense above: they are two documents, both current as documents, that disagree.

Research on how models handle this is sobering. A benchmark of over 1,200 questions with deliberately altered retrieved content found that “LLMs are susceptible to adopting incorrect retrieved content, overriding their own correct prior knowledge over 60% of the time”, while the more the retrieved content departed from the truth, the less likely the model was to adopt it (Wu et al., ClashEval). Another study asked what a model should do when its prompt contains conflicting information, and found that models “perform well in identifying the existence of knowledge conflicts” but “struggle to determine the specific conflicting knowledge and produce a response with distinct answers amidst conflicting information” (Wang et al., Resolving Knowledge Conflicts in Large Language Models). The experience report adds that failure to extract the right answer “typically” occurs with “too much noise or contradicting information in the context”. Read together, they say not to make the model the referee.

The design that follows resolves the conflict in code, using the edit date you stored for each document: when the retrieved chunks give different values for the same thing, keep the chunks from the document edited most recently and drop the rest. This is the lever prefer the newest source. A version number cannot do this job, because each document counts its own versions and a 3 in one document says nothing about a 1 in another; a date can be compared across documents. It is a policy, and it has limits: it works only when the newer document really supersedes the older one. When two documents cover different cases, a general rule and a travel exception, the right design is to keep both and label them, which is a metadata and prompt decision.

A fix is a change: re-measure everything

Apply the cheapest repair for Ticket 3, a larger k, to the whole system. Now every question retrieves one more chunk. In the project’s data, a question about how old a receipt can be, which was answered correctly before, now retrieves a chunk from a travel document that says 90 days beside one from the general policy that says 60. The change repaired Ticket 3 and created a conflict for another question that had nothing to do with it.

That is a regression, and it is the reason the habit is to keep known cases and re-run all of them after every change. It matches the report’s own lessons: “RAG systems require continuous calibration”, and “testing performance characteristics are only possible at runtime”. Each fix is a change to a system whose behaviour depends on all its settings, so a fix is accepted only when the whole set of known cases still passes, or the new failures are understood and chosen.

Three situations you will meet, and what they mean for your design

Before the table, three situations:

1. The handbook is edited on Tuesday. Nothing is re-ingested. On Friday a customer quotes the assistant’s old answer back at a support agent who is reading the new text.
2. A team widens k from 3 to 8 to fix a handful of “not found” tickets. A week later, tickets appear in which the assistant mentions two different values in one answer.
3. A policy is replaced by a new document, but the old document stays in the folder that is ingested.

What you see	What failed	What to do instead
A confident, cited, outdated answer (situation 1)	The index drifted from the live document	Store a source version with each chunk, compare it with the live version, and re-ingest on a mismatch
New conflicts after widening k (situation 2)	A fix with a side effect; the other cases were not re-run	Keep known cases and re-run all of them after every change
Two authorities, two values (situation 3)	The old source was never retired	Retire the old document, or keep an edit date on each and prefer the newest source in code

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

What is a stale index, and why do the citation checks from Chapter 3 not catch it?

An index built from an older version of a document than the live one. The citation checks compare an answer with the stored chunk, and the chunk is what is out of date, so a correct pointer and a matching quote can carry an old answer.

How do you check whether a chunk is stale?

Store the source version with each chunk when it is built, and compare it with the live version of the document. A mismatch means stale, and the repair is to re-ingest the document. A chunk with no recorded version cannot be checked this way.

What does the diagnosis ladder do, and in what order does it check?

It runs cheap checks in a fixed order and names the first that fails: is the answer in the live documents, is the index current, does a chunk hold the whole answer, is that chunk in the top k, do the retrieved chunks agree, and if all pass, the fault is after retrieval.

Why does the ladder check the live documents before it checks retrieval?

A text comparison against the live documents costs almost nothing and rules out missing content. A retrieval re-run costs more and tells you nothing if the answer was never in the corpus.

What is a vocabulary mismatch, and how does it cause a retrieval miss?

The question and the passage that answers it use different words for the same thing, such as connect from home and remote staff reach. A score based on shared words sees little overlap, so the right chunk ranks low or ties with others and falls outside the top k.

What does the report's second failure point say?

The answer is in the document but did not rank highly enough to be returned, because in practice only the top K documents are returned, with K chosen on performance.

What did the ClashEval benchmark find about incorrect retrieved content?

That the models tested adopted incorrect retrieved content, overriding their own correct prior knowledge, over 60% of the time. The more the content departed from the truth, the less likely the model was to adopt it. Those numbers are for that study’s models and data.

Why not let the model decide between two conflicting sources?

A study of knowledge conflicts found that models identify that a conflict exists but struggle to pinpoint the conflicting information and to answer with distinct answers. Resolving it in code, with a stored edit date, does not depend on that.

What is a regression, and why does it follow some fixes?

A case that worked before a change and fails after it. A fix such as a larger k changes what every question retrieves, so it can repair one case and break another that was unrelated.

Which repair ("lever") goes with a split answer, and which with a retrieval miss?

A split answer is repaired by re-chunking, with a larger size, an overlap or sentence boundaries. A retrieval miss is repaired first by raising k, and then by changing what is matched on, for example by adding a dense or hybrid search.

Model answer

The question: “Users report wrong answers from our document assistant in production. How do you find out why and fix it without making things worse?” This is the question a lead engineer asks after an incident, and an interviewer for a role building with models asks to hear a debugging routine. A strong answer hits five beats, in this order:

1. Reproduce — turn each report into a known case: the question and the wording that should answer it.
2. Ladder — run cheap checks in a fixed order: in the documents, index current, answer intact in a chunk, chunk in the top k, sources agreeing.
3. Lever — apply the repair that matches the first failing check: add a document, re-ingest, re-chunk, raise k, or prefer the newest source.
4. Re-measure — re-run every known case, not only the one that was reported, and look for regressions.
5. Monitor — keep the cases, a source version on every chunk and an edit date on every document, because production brings new failures after launch.

“First I’d turn each report into a known case: the question, and the exact wording in the live documents that should answer it. Then I’d run a ladder of cheap checks in order and stop at the first that fails. Is the wording in the live documents at all? If not, it’s missing content, and the fix is a document, not a model change. Is the index built from the current version of that document? A stored source version makes that a comparison of two numbers, and a mismatch means re-ingest. Is the answer inside one chunk? If not, it was split, and I’d re-chunk. Is a chunk holding it among the top k? If not, the search missed it, maybe a vocabulary mismatch, and the first lever is a larger k. Do the retrieved chunks agree? If two sources give different values, I’d resolve it in code using the documents’ edit dates rather than ask the model to referee. After any fix I’d re-run all the known cases, because a larger k, for instance, can create a conflict in a question that was fine. And I’d keep those cases, the versions and the edit dates in place, because the research I’ve read says validation is only really possible during operation and robustness evolves after launch.”

Missing a beat is a failure state. Without reproduce there is nothing to measure. Without ladder the fix is a guess at a stage. Without lever the diagnosis changes nothing. Without re-measure a repair for one ticket can break another unseen. Without monitor the same failures return.

Why this matters

• Documents change after launch. Policies are edited, products are renamed and teams retire pages. A retrieval system without a source version on each chunk has no way to tell that its answers have aged.
• Support tickets become test cases. Each reported failure, written as a question and an expected wording, is a case you keep. A set that grows from real tickets reflects what users actually ask.
• Fixes interact. Chunk size, overlap, k and the conflict policy all act on the same pipeline, so a change should be accepted on the whole set of cases, not on the one it was made for.
• Talking to customers. When a customer says “it told me the old rule”, an engineer who can say “the index was built before that edit, and here is the check that now prevents it” gives a cause and a guard.

After this chapter

The project for this chapter is Diagnose Six Failures, Fix Them, and Re-measure (project.md). You will write a diagnosis ladder and run it on six failing tickets, apply four fixes in order and re-run all six after each stage to see what each repaired and what each broke, and finish by writing causes.tsv: each ticket’s cause and the lever that repairs it.

A later lesson in Getting Started with AI opens with that file. It takes the cause behind each of your tickets and asks which would be repaired by changing the model’s behaviour, and which by changing the data and the pipeline it reads from.

Hands-on Project — Diagnose Six Failures, Fix Them, and Re-measure

Objective

Write a diagnosis ladder that names the first stage at which each of six reported questions lost its answer, if it did, apply four fixes one at a time, and re-run the ladder after each change, so that you can show with counts which tickets a fix repaired and which it broke.

What this project needs: Python 3 and its standard library only (re). No task needs an account with a model provider, an API key, a vector database or any payment. The documents, the index and the six tickets are printed below, and every check is a string comparison or a count, so every result can be checked exactly. All three tasks are required.

Your evidence file. Keep one plain-text file, chapter4-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print. Where a value is a list, write its items comma-separated with no spaces, in the order given.

The data and the rules for all tasks

Copy this into your Python file. LIVE holds the documents as they read today, each as (version, date last edited, text). INDEX holds the chunks the search currently serves: (chunk id, document, version the chunk was built from, chunk text). A ticket is (ticket id, question, expected wording, k, pattern): the expected wording is the text that answers the question in the live documents, k is how many chunks the pipeline retrieves, and pattern is a regular expression with one group that picks out the value a chunk states for the thing the question asks about.

import re

LIVE = {
    "handbook": (3, "2026-09-12", "A loan laptop must be returned within 14 days of the issue date. Staff who need a laptop for longer must ask their manager to approve an extension."),
    "faq": (1, "2025-03-02", "A loan laptop must be returned within 30 days of the issue date."),
    "wifi": (2, "2026-08-03", "The guest wifi password changes on the first Monday of each month."),
    "network": (1, "2026-02-20", "Staff devices must use the corporate network instead of the guest network. Remote staff reach the corporate network through the VPN."),
    "expenses": (1, "2026-01-15", "Claims below 50 dollars do not need a receipt. Claims of 50 dollars or more must include a receipt image. Receipts older than 60 days will be rejected by the portal."),
    "travel": (1, "2026-01-15", "A receipt older than 90 days is accepted for travel claims."),
}

INDEX = [
    ("handbook-1", "handbook", 3, "A loan laptop must be returned within 14 days of the issue date."),
    ("handbook-2", "handbook", 3, "Staff who need a laptop for longer must ask their manager to approve an extension."),
    ("faq-1", "faq", 1, "A loan laptop must be returned within 30 days of the issue date."),
    ("wifi-1", "wifi", 1, "The guest wifi password changes every Monday."),
    ("network-1", "network", 1, "Staff devices must use the corporate network instead of the guest network."),
    ("network-2", "network", 1, "Remote staff reach the corporate network through the VPN."),
    ("expenses-1", "expenses", 1, "Claims below 50 dollars"),
    ("expenses-2", "expenses", 1, "do not need a receipt. Claims of 50 dollars or more must include"),
    ("expenses-3", "expenses", 1, "a receipt image. Receipts older than 60 days will be rejected by the portal."),
    ("travel-1", "travel", 1, "A receipt older than 90 days is accepted for travel claims."),
]

TICKETS = [
    ("t1", "Can I bring my dog to the office?", "dogs are allowed in the office", 2, r"(\d+) dogs"),
    ("t2", "When does the guest wifi password change?", "on the first Monday of each month", 2, r"changes (.+?)\."),
    ("t3", "Do I need a receipt for a 30 dollar expense claim?", "Claims below 50 dollars do not need a receipt", 2, r"below (\d+) dollars"),
    ("t4", "Where do staff connect from home?", "Remote staff reach the corporate network through the VPN", 2, r"reach the (.+?) through"),
    ("t5", "How many days can a loan laptop be kept before it must be returned?", "within 14 days of the issue date", 2, r"within (\d+) days"),
    ("t6", "How old can a receipt be before the portal rejects it?", "Receipts older than 60 days will be rejected by the portal", 2, r"older than (\d+) days"),
]

STOP = set("the a an of to is are in on for be by and or it that this who which what how do i can will must from at with as when".split())

def words(text):
    return re.findall(r"[a-z0-9]+(?:-[a-z0-9]+)*", text.lower())

def norm(text):
    # lower-cased words joined by single spaces: ignores case and punctuation
    return " ".join(words(text))

def score(question, chunk_text):
    # number of different non-stop words that the question and the chunk share
    return len((set(words(question)) - STOP) & (set(words(chunk_text)) - STOP))

The retrieval rule. Rank every chunk in the index by score(question, chunk text), highest first; when two chunks have the same score, the smaller chunk id (as text) goes first. Drop chunks with a score of 0. Keep the first k.

The diagnosis ladder. For a ticket, take the first of these that applies:

1. NOT_IN_CORPUS — norm(expected) appears inside norm(text) of no live document.
2. STALE — for a live document whose text contains the expected wording, some index chunk of that same document was built from a version different from the document’s live version (the first entry of its LIVE tuple).
3. SPLIT — no index chunk’s text contains the expected wording (norm(expected) inside norm(chunk text)).
4. MISSED — some chunk contains the expected wording, but none of those chunks is among the kept chunks.
5. CONFLICT — apply re.search(pattern, chunk text) to each kept chunk; the value is group 1 of a match, and a chunk without a match gives no value. Two or more different values among the kept chunks is a conflict.
6. OK — none of the above applies: the answer text reached the retrieval stage intact and unchallenged. Any fault is later, in how the model is asked to use it.

Task 1 — Build the ladder and diagnose six tickets

Write diagnose(ticket, index, live, k=None, newest=False), which returns one of the six status names following the ladder. When k is given, it replaces the ticket’s own k. Leave newest unused in this task (Task 2 defines it). Print the status of all six tickets.

Add to your evidence file:

t1_t1: <status of t1>
t1_t2: <status of t2>
t1_t3: <status of t3>
t1_t4: <status of t4>
t1_t5: <status of t5>
t1_t6: <status of t6>
t1_ok_count: <number of tickets with status OK>

Output: one Python file with a diagnosis function, and seven evidence lines.

Task 2 — Apply four fixes and re-measure

This task stands on its own: it needs the data and rules above and a diagnose function, which you can copy from Task 1 or write again from the ladder.

Apply these fixes one after another, in order, re-running all six tickets after each fix, so that you can see what each one repaired and what each one broke. F3 is applied on top of F1 and F2, and F4 on top of F1, F2 and F3.

• F1, re-ingest the wifi document. In INDEX, replace the chunk wifi-1 with ("wifi-1", "wifi", 2, <the live text of wifi>).
• F2, re-chunk the expenses document. Remove every expenses-* chunk from INDEX. Chunk the live text of expenses with the chunking rule below, and add the chunks to INDEX as (chunk id, "expenses", 1, chunk text), with chunk ids expenses-1, expenses-2 and so on.
• F3, raise k. Call diagnose with k=3 for every ticket.
• F4, prefer the most recently edited source. Add the newest=True behaviour to diagnose: at step 5, when the kept chunks that match the pattern show two or more different values, find the latest date last edited (the second entry of the LIVE tuple) among the documents of those matching chunks, drop the matching chunks whose document was last edited on an earlier date, and count the different values among what is left. Dates are written YYYY-MM-DD, so they compare correctly as text.

The chunking rule. Split the text into words with text.split(). A chunk is 30 words joined by single spaces. The first chunk starts at word 0, and each next chunk starts 20 words after the previous one (a size of 30 and an overlap of 10). Stop after the first chunk whose end reaches or passes the last word. Chunks are named <document>-<n>, counting from 1.

Print the six statuses after F1, after F1 and F2, after F1 to F3, and after F1 to F4.

Add to your evidence file:

t2_after_f1: <the six statuses after F1, in ticket order>
t2_after_f12: <the six statuses after F1 and F2, in ticket order>
t2_after_f123: <the six statuses after F1, F2 and F3, in ticket order>
t2_after_f4: <the six statuses after F1, F2, F3 and F4, in ticket order>
t2_ok_counts: <the number of OK tickets after F1, F1 to F2, F1 to F3 and F1 to F4, comma-separated>
t2_regressed_f3: <ids of tickets that were OK before F3 was applied and are not OK after it, or none>
t2_still_failing: <ids of tickets not OK after all four fixes, in ticket order>

Before you submit, compare each ticket’s result before and after the fixes. If any ticket got worse or stayed in conflict, write one sentence for yourself on why the same fix can help one ticket and hurt another; it is not scored.

Output: the four fixes applied, and seven evidence lines.

Task 3 — Write the causes file

This task stands on its own: it needs the data above and a diagnose function, as in Task 1.

Using the original INDEX, LIVE and each ticket’s own k (no fixes applied), write causes.tsv: one line per ticket, in order, with three fields separated by single tab characters: the ticket id; the status from the ladder; and the lever that status maps to in this table.

Status	Lever
NOT_IN_CORPUS	add-document
STALE	re-ingest
SPLIT	re-chunk
MISSED	raise-k
CONFLICT	prefer-newest
OK	none

Add to your evidence file:

t3_rows: <number of lines in causes.tsv>
t3_levers: <the third field of every line, in file order, comma-separated>

Output: a causes.tsv of six lines, and two evidence lines.

What carries forward

Keep causes.tsv and chapter4-evidence.txt. A later lesson in Getting Started with AI opens with causes.tsv: it takes the cause behind each of your six tickets and asks which of them would be repaired by changing the model’s behaviour, and which by changing the data and pipeline the model reads from.