Chapter 2 — Retrieval-Augmented Generation End to End

Course: Build and Evaluate RAG Applications

Prerequisites: You can read and run a short Python script. From Chapter 1 you need the idea that a search returns the passages whose vectors are closest to a query, that this closeness measures relatedness rather than correctness, and that recall at k counts how often the passage that answers a question reaches the top k. The notes restate each of these in a sentence before they use it. Chunking, indexing, retrieval, assembly and context budget are all defined in the notes before they are used.

What the reader can do after this chapter:

• Split documents into chunks with a stated size and overlap, and show which questions have their answer cut in half by a given setting.
• Trace one question through the four stages of a retrieval-augmented pipeline (chunk, index, retrieve, assemble) and name the stage at which its answer was lost.
• Choose the chunk size, overlap, number of retrieved chunks and context budget for a small corpus by measuring how many known questions keep their answer at each stage.

Core question this chapter answers: When a retrieval-augmented assistant gives a wrong or empty answer, at which of the pipeline’s stages did the answer text get lost, and what does each stage’s setting trade away?

Connections:

• Builds on: Chapter 1, which measured a single stage, vector search, with a set of questions whose answers you know. This chapter puts that stage inside a whole pipeline and measures every stage with the same kind of question set, shaped like the gold.tsv file you wrote in Chapter 1’s project.
• Leads into: Chapter 3, which makes each claim in a generated answer point back to a source chunk, and makes the system refuse when no chunk supports an answer. It opens with the chunks.tsv file your project writes here.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question, so you can check that your answer has the right shape before you submit it.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines. Every line’s accepted value, or the rule that decides it, follows from what the task prints, so you can check your own file before you submit it.

Build and Evaluate RAG Applications — Chapter 2: Retrieval-Augmented Generation End to End

Before you start

Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library. You do not need an account with any model provider or an API key: the project works on a small corpus printed in the project file and never calls a model.

Keep one question in mind while you read: when the assistant’s answer is wrong, where did the right text drop out on its way to the model? By the end of the chapter you should be able to point at a stage and a setting, and back the claim with a count.

Summary

A retrieval-augmented assistant does not answer from memory alone: it looks the answer up in your documents and hands what it found to the model. That lookup is a pipeline of four stages. Chunking cuts each document into pieces, indexing stores the pieces so they can be searched, retrieval picks the pieces that match a question, and assembly places the picked pieces into the model’s prompt within a size budget. Each stage has a setting, and each setting gives something up: small chunks lose their surroundings, large chunks blur what they are about, a larger number of retrieved pieces costs room in the prompt, and a smaller one risks missing the piece that answers. An answer can be lost at any stage, and the model cannot recover text that never reached its prompt, so the engineering habit is to measure each stage with the same set of questions whose answers you already know, so that a wrong answer points to a stage instead of to “the model”.

Key concepts

Term	Definition
Retrieval-augmented generation (RAG)	Generating an answer with a model that is given passages looked up from a separate store of documents, instead of relying only on what the model absorbed in training.
Chunk	One piece of a document, cut out so it can be indexed and retrieved on its own. Its size can be counted in tokens or words.
Chunk size	The most text one chunk may hold, counted in tokens or words.
Chunk overlap	The amount of text that two neighbouring chunks share, so that a sentence cut at one chunk’s end is whole at the start of the next.
Embedding	A fixed-length list of numbers that an embedding model produces for a piece of text, so that texts about related things get lists that are close together (Chapter 1).
Vector search	Finding the stored chunks whose embeddings are closest to the question’s embedding, usually by cosine similarity (Chapter 1).
Lexical search	Ranking chunks by the words they share with the question, so an exact string such as an error code is found by its text (Chapter 1).
Hybrid search	Running a lexical and a vector ranking side by side and merging them into one list with a rule such as reciprocal rank fusion, which adds a score that falls with rank from each list (Chapter 1).
Approximate search	Vector search that compares the question against fewer stored vectors, for speed or memory, at some chance of missing a true nearest chunk (Chapter 1).
Index	The stored form of the chunks that a search runs against: for vector search, each chunk’s embedding plus the chunk text and any attached fields such as its source document.
Retrieval	Selecting, for one question, the top k chunks from the index. The number k is a setting.
Assembly	Building the prompt from the retrieved chunks and the question, within a limit on how much text may be passed to the model.
Context budget	The amount of text, in tokens or words, that you allow retrieved chunks to occupy in the prompt. It is smaller than the model’s window because the instructions, the question and the reply need room too.
Stage	One step of the pipeline: chunking, indexing, retrieval or assembly.
Intact answer	An answer whose full wording sits inside one single chunk, rather than being cut across two chunks.
Gold set	A set of questions, each paired with the passage that answers it, used to measure retrieval. Chapter 1’s gold.tsv is one.
Stage-wise recall	For each stage, the share of gold questions whose answer is still intact (whole, inside one chunk) among the chunks that stage passes on. Under that counting rule it can stay the same or fall from one stage to the next, because each stage passes on a subset of the chunks it received.

Explanations

The failure, before the explanation

Here is an assistant that answers questions about a company’s internal policies. Its documents were cut into chunks of twelve words with no overlap, and for each question only the single best chunk is passed to the model. A user asks:

Question: “Do I need a receipt for a 30 dollar expense claim?”

The chunk the model was given: “do not need a receipt. Claims of 50 dollars or more must”

Reply: “Claims of 50 dollars or more must include a receipt, so for 30 dollars you should check with the finance team.”

The policy says the opposite of “check with the finance team”: its sentence is “Claims below 50 dollars do not need a receipt.” But the sentence was cut in two. Its first half, “Claims below 50 dollars”, sits at the end of the previous chunk, and the chunk the model was given starts in the middle of the answer. The model did its best with half a sentence. Nothing crashed, nothing looks broken in the logs, and the reply reads as polite caution.

An engineer who has debugged services will want to blame the model first. The model is not where this went wrong: the right words were never in its prompt in one piece. To see where, you need the shape of the pipeline.

The pipeline: an open-book exam with a librarian

Picture an open-book exam. The student (the model) can reason and write, but the facts have to come from the book. A librarian (the retrieval pipeline) hears the question, runs to the shelves, and puts a few pages on the student’s desk. If the page with the answer is not on the desk, the student can still write something fluent, and it will be a guess.

That picture matches the research paper that gave retrieval-augmented generation its name, which describes models that “combine pre-trained parametric and non-parametric memory for language generation”, where the parametric memory is a pre-trained language model and the non-parametric memory is, in that paper, “a dense vector index of Wikipedia, accessed with a pre-trained neural retriever” (Lewis et al., Retrieval- Augmented Generation for Knowledge-Intensive NLP Tasks). In plain terms: what the model learned in training is one memory, and a searchable store of documents is a second one that the model consults.

A second way to picture it: a build pipeline. Source goes through compile, test and package before it is deployed. A defect can enter at any step, and the deployed artifact shows only that something is wrong, not where. You find the step by checking the output of each one. A retrieval pipeline works the same way, with four steps:

1. Chunk — cut each document into pieces.
2. Index — store the pieces in a form that can be searched.
3. Retrieve — for a question, pick the top k pieces.
4. Assemble — put the picked pieces and the question into the prompt, within a budget.

The model’s own generation comes after these four and is the subject of the later chapters. This chapter is about the four stages that decide what the model gets to read.

Stage 1, chunking: where the rope is cut

Start with the failure above. A twelve-word chunk of the expenses policy begins “do not need a receipt.” The sentence that answers the question is “Claims below 50 dollars do not need a receipt”. A chunker that cuts every twelve words has no idea where sentences begin, so the cut fell inside the answer. A chunk whose full answer is inside it is an intact answer; this one is not.

A second way to see it: grep with no context lines. If you search a log for ERROR and print only the matching line, you often lose the line before it that said what was running. Printing a few lines either side costs a few more lines on screen and keeps the match readable. Chunk overlap is that same trade: you repeat some text at the edges so that a cut does not strand half a sentence.

There are two settings, and each gives something up.

Chunk size. Two sources describe the pull in each direction. One vector-database vendor’s guide says that “if our chunks are too small or too large, it may lead to imprecise search results or missed opportunities to surface relevant content”, and offers a rule of thumb: “if the chunk of text makes sense without the surrounding context to a human, it will make sense to the language model as well” (Pinecone, Chunking Strategies for LLM Applications). A model provider gives a concrete case of a chunk that is too small to stand alone. A chunk reading “The company’s revenue grew by 3% over the previous quarter.” “on its own doesn’t specify which company it’s referring to or the relevant time period” (Anthropic, Introducing Contextual Retrieval). The sentence is intact and still unusable, because the words that say which company and which quarter were left in another chunk.

Chunk overlap. With no overlap, a cut can strand half an answer, as above. With overlap, the end of one chunk is repeated at the start of the next. The cost is that you now store, embed and may retrieve the same text twice. One provider’s vector-store documentation gives its defaults: “By default, max_chunk_size_tokens is set to 800 and chunk_overlap_tokens is set to 400, meaning every file is indexed by being split up into 800-token chunks, with 400-token overlap between consecutive chunks”, adjustable with a chunking_strategy setting, where the size must be between 100 and 4096 tokens and the overlap “should not exceed max_chunk_size_tokens / 2” (OpenAI, Retrieval). A default is one provider’s choice for its own service, not a rule for your corpus. Another provider’s write-up says only that “the choice of chunk size, chunk boundary, and chunk overlap can affect retrieval performance” (Anthropic, Introducing Contextual Retrieval). That sentence is the honest summary: there is no universal best number, and you find yours by measuring.

The project’s corpus (three short policy documents and six questions about them) makes the effect visible. Chunked at twelve words with no overlap, the six questions have 4 of 6 answers intact; chunked at thirty words with ten words of overlap, all 6 are. The cost of the second setting is real too: it makes fewer chunks (9 against 17), but each repeats text from its neighbour, so the same words are stored and can be retrieved twice.

Stage 2, indexing: the card catalogue

Once the chunks exist, they have to be findable. Think of a library: books on shelves are the chunks, and the card catalogue is the index. Nobody searches by walking every shelf; they search the catalogue. In a vector store, the catalogue entry for a chunk is its embedding (Chapter 1), stored beside the chunk’s text and any fields you attach. One provider describes the step as automatic: “When you add a file to a vector store it will be automatically chunked, embedded, and indexed” (OpenAI, Retrieval). Chunking, embedding and indexing are one ingestion step in that service, which is why a chunking setting is easy to change in code and expensive to change in practice: changing it means re-ingesting the files.

A second way to see it: a database index. Adding an index to a table costs storage and slows writes, and in return reads become fast. A vector index makes the same bargain, and Chapter 1’s approximate search, which compares a question against fewer stored vectors, is the speed-for-recall version of it. Two things from Chapter 1 carry over without change. An embedding is a list of numbers that a model produces for a text, so that related texts get nearby lists. The embedding model and its version are part of the index, so a new model means re-embedding every chunk. And a lexical index (one that ranks chunks by the words they share with the question) can sit beside the vector index, so that an exact error code can be found by its text.

What this stage adds to the pipeline is fields: the source document’s name, the section, the date, stored beside each chunk. A field stored at indexing time is what lets assembly put a source name back into the prompt later; a field that was never stored cannot be added at that point. Chapter 3 depends on this.

Stage 3, retrieval: how many pieces to pull

Take the project’s sixth question, “How old can a receipt be before the portal rejects it?”. The answer sits in one chunk, expenses-2. Two chunks score the same on the project’s word-matching rule, and the tie is broken by chunk id, so expenses-1 ranks first. If only the top chunk is kept (k = 1), the answer chunk is not returned; if the top two are kept (k = 2), it is. Retrieval recall rose from 5 of 6 questions to 6 of 6, and the only change was k.

That is the whole stage. Retrieval takes the question, scores the chunks, and returns the top k. The setting is k, and the pull is between two losses. A small k risks leaving out the chunk that answers. A large k fills the prompt with chunks that do not.

A second way to think about it: shortlisting candidates for an interview. Interview one and you may miss the best person, who ranked second on the CV screen. Interview fifty and the process takes forever. The right shortlist length comes from how often the best candidate really sits in positions one to five, which is a measurement.

Real services expose the same dials. One vendor’s documentation says a retrieval response “will contain 10 results maximum by default, but you can set up to 50 using the max_num_results parameter”, and that a score_threshold between 0.0 and 1.0 limits results to more relevant chunks, “though it may exclude some potentially useful ones” (OpenAI, Retrieval). A cutoff on score is a second way to choose how many pieces to pull, and it has the same trade: a stricter cutoff excludes more of the wrong pieces and some of the right ones. From Chapter 1, retrieval can also run a lexical and a dense ranking and merge them into one list, which is hybrid search; the same vendor exposes weights for that merge.

Stage 4, assembly: what fits on the desk

Stay with the sixth question. Retrieval, with k = 3, hands over three chunks of 30 words each, expenses-1 first and expenses-2 second. The prompt is allowed 50 words of retrieved text. The first chunk fits, bringing the total to 30. The second would bring it to 60, so it is left out, and so is the third. The answer was in the second chunk. Retrieval did its job and the answer still never reached the model: it was lost at assembly.

That is the stage. Assembly turns the ranked list into a prompt, and the prompt has a limit. The context budget is the amount of text you allow retrieved chunks to take, and it is smaller than the model’s window because the instructions, the question and the model’s reply need room too.

A second example: packing a suitcase with a weight limit. The best items are chosen first, in order of priority, and when the next item would push the bag over the limit, it stays at home however useful it was. The order you pack in decides what is left behind.

Assembly has three decisions beyond the budget, each with a published basis.

• Order. One provider’s guidance for long inputs says to “place your long documents and inputs near the top of your prompt, above your query, instructions, and examples”, and that “queries at the end can improve response quality by up to 30 percent in tests, especially with complex, multidocument inputs” (Anthropic, Long context prompting tips). That figure is the provider’s own test result, not a promise for your prompt.
• Labelling. The same guidance suggests that “when using multiple documents, wrap each document in <document> tags with <document_content> and <source> (and other metadata) subtags for clarity”. The source subtag is where the fields from the indexing stage return.
• Whether to retrieve at all. One provider’s write-up says: “If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods” (Anthropic, Introducing Contextual Retrieval). That figure is the one the provider states; check the limit of the model you actually use.

The pipeline as a whole: counting what survives

Take the project’s six questions, with thirty-word chunks, ten words of overlap, k = 3 and a budget of 50 words, the counts are:

After this stage	Questions whose answer is still present
Chunking (answer intact in a chunk)	6 of 6
Indexing (the chunk is stored and can be found)	6 of 6
Retrieval (answer chunk in the top 3)	6 of 6
Assembly (answer chunk fits in the budget)	5 of 6

The project has no separate index: retrieval scores the chunk list directly, so the indexing row cannot lose anything there. In a real service you check it with two counts. First, compare the number of chunks your chunker produced with the number the store reports, because a file that failed to ingest silently drops all of its chunks. Second, probe: take a distinctive phrase from a chunk’s text, search for it, and confirm that the chunk comes back. A question whose answer chunk fails the probe was lost at indexing, and the fix is in ingestion, not in k or the budget.

Read down the column and the numbers stay level or fall. That follows from the counting rule used here: a question counts as present only while its whole answer sits inside one chunk that the stage passes on, and each stage passes on some or all of the chunks it received, never new ones. So under this rule, an answer that is not intact in any chunk after chunking is not counted at retrieval or assembly either. A real system could in principle count an answer rebuilt from two adjacent fragments that are both retrieved; this project does not, so that case is outside the count.

Each count is measured against the same gold set, which is the same idea as Chapter 1’s gold.tsv: a question paired with the passage that answers it. The sixth question is the one that was lost, and the table says where. Without a table like this, the report would be “the assistant got one wrong”, with no stage to blame and no setting to change.

This also tells you what a correct answer from the model does not prove. If the answer chunk reached the prompt and the reply is still wrong, every retrieval stage did its job and the fault is later, in how the prompt asks the model to use what it was given. Equally, a right answer can come from the model’s own training, which is why the counts above are counted on the text that reached the prompt and not on the replies.

Three situations you will meet, and what they mean for your design

Before the table, three situations:

1. A support assistant answers “can’t tell from the documents” about a rule the policy states in one clear sentence. Searching the chunk file for the sentence finds half of it at the end of one chunk and half at the start of the next.
2. A team raises k from 3 to 10 to catch more answers. Replies get slower and more expensive, and several of them start quoting the wrong document.
3. The right chunk is retrieved and appears in the logs, but the assistant answers as if it had not seen it. The prompt was built by concatenating chunks until a size limit, and the chunk was the one that got cut off.

What you see	Which stage lost the answer	What to do instead
Half a sentence in two chunks (situation 1)	Chunking: no chunk holds the whole answer	Add overlap or cut on sentence boundaries; re-measure intact answers
More chunks, worse replies (situation 2)	Generation: the right chunk is still there, now among many wrong ones; if the budget cut it, assembly	Measure recall at several k; set k from the gold set, not by guess
Retrieved but ignored (situation 3)	Assembly: the chunk did not fit the budget	Count tokens per chunk, log what was included, and set the budget on purpose

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

What are the four stages that decide what the model reads in a retrieval-augmented pipeline?

Chunk (cut documents into pieces), index (store the pieces so they can be searched), retrieve (pick the top k pieces for a question) and assemble (put the picked pieces and the question into the prompt within a budget).

What does chunk overlap do, and what does it cost?

It repeats the end of one chunk at the start of the next, so a sentence cut at a chunk’s edge is whole in the next chunk. It costs storage and embedding work, and the same text can be retrieved twice.

Why can a chunk that is too small fail even when its sentence is intact?

It can lose the words that say what it refers to. One provider’s example is “The company’s revenue grew by 3% over the previous quarter”, which does not name the company or the period on its own.

What are one provider's default chunk size and overlap for its vector stores?

800 tokens per chunk with 400 tokens of overlap between consecutive chunks, adjustable. This is one service’s default, not a rule for every corpus.

What is the trade in choosing k, the number of chunks retrieved?

A small k risks leaving out the chunk that answers, which lowers recall. A large k fills the prompt with chunks that do not answer, which costs budget and can bury the right one.

What is a context budget, and why is it smaller than the model's window?

The amount of text you allow retrieved chunks to occupy in the prompt. It is smaller than the window because the instructions, the question and the model’s reply need room in the same window.

Retrieval returned the answer chunk, but the model never used it. What do you check?

Whether the chunk actually reached the prompt. If the budget cut it off during assembly, the answer was lost there, even though retrieval was right.

Where does one provider's guidance say to put long documents and the query in a prompt?

Long documents near the top, above the query, instructions and examples, and the query at the end. The provider reports up to a 30 percent quality gain in its own tests for complex, multi-document inputs.

When does one provider say you can skip retrieval entirely?

When the knowledge base is smaller than 200,000 tokens, about 500 pages, you can include all of it in the prompt. That figure is that provider’s, for its models, so check the limit of the model you use.

Why does stage-wise recall stay level or fall in this project's counting?

Each stage passes on a subset of the chunks it received, and a question counts only while its whole answer is inside one of those chunks. A question whose answer was not intact after chunking therefore has no chunk that could satisfy the test at retrieval or assembly, so the count stays level or falls. A counting rule that also accepted an answer rebuilt from two adjacent fragments would be a different rule.

Model answer

The question: “Walk me through a retrieval-augmented pipeline for our policy documents, and tell me how you would find out where it loses answers.” This is the question a lead engineer asks before approving the design, and an interviewer for a role building with models asks to hear a debugging habit. A strong answer hits five beats, in this order:

1. Chunk — documents are cut into pieces, with a size and an overlap chosen against the risk of cutting an answer in half.
2. Index — the chunks are embedded and stored with their source fields, and the embedding model’s version is treated as part of the index.
3. Retrieve — for a question, the top k chunks are selected, and k trades missing the answer against crowding the prompt.
4. Assemble — the chunks go into the prompt within a context budget, ordered and labelled, with the question last.
5. Measure — a gold set of questions with known answers is counted at each stage, so a lost answer is blamed on a stage and a setting.

“I’d treat it as four stages that decide what the model gets to read. First, chunking: I’d cut the documents with a size and an overlap, because a cut can land in the middle of the sentence that answers a question, and then no chunk holds the whole answer. Second, indexing: I’d embed the chunks and store them with the source document and section as fields, and I’d pin the embedding model’s version with the index. Third, retrieval: I’d pick the top k chunks, where a small k risks missing the chunk that answers and a large k crowds the prompt with chunks that don’t. Fourth, assembly: I’d place the chunks in the prompt within a budget that leaves room for the instructions and the reply, with the source labelled and the question last. To find out where it loses answers I would not read replies and guess. I’d keep a gold set of real questions, each paired with the passage that answers it, and count, for each stage, how many answers are still present. With an answer counted only when it is whole inside one chunk, the count can’t rise from stage to stage, so the stage where the count drops is the one to fix, and I’d re-run the same set after changing that setting.”

Missing a beat is a failure state. Without chunk the answer cannot explain a half-sentence failure. Without index it cannot say why the model version and the source fields matter. Without retrieve it cannot say what k trades. Without assemble it cannot explain an answer that was retrieved and still missing from the prompt. Without measure every fix is a guess.

Why this matters

• Grounded answers are built on it. The next chapter makes each claim in an answer point at a source chunk. A chunk that holds only half a rule cannot be cited for the whole rule, and a source name that was never stored cannot be shown.
• Cost and speed follow the settings. Every retrieved chunk is text the model has to read, so k, chunk size and budget set how much text each request carries, which the cost and latency numbers in Getting Started with AI already count.
• Changes are deployments. A new chunk size means re-ingesting the documents, and a new k or budget changes what the model sees. Each deserves the same before-and-after count as any release.
• Talking to customers. When a customer says “it ignored the policy”, an engineer who can show that the sentence was cut across two chunks, or that it did not fit the budget, can give a cause and a fix instead of an apology.

After this chapter

The project for this chapter is Count What Survives Each Stage (project.md). You will chunk a small corpus with two different settings and count the answers each one keeps intact, retrieve with different values of k and count what the retrieval stage keeps, and assemble prompts under a budget to find the question whose answer is lost there. You finish by writing the chunks to a file that the next chapter starts from.

Chapter 3 opens with that file. It takes your chunks and makes a generated answer cite the chunk each claim came from, and teaches the system to refuse when no chunk supports an answer.

Hands-on Project — Count What Survives Each Stage

Objective

Chunk a small corpus with two settings, retrieve from it with different numbers of chunks, assemble prompts under a budget, and count after each stage how many known questions still have their answer present, so that you can name the stage and the setting behind a lost answer instead of blaming the model.

What this project needs: Python 3 and its standard library only (re). No task needs an account with a model provider, an API key, a vector database or any payment. The corpus is printed below and the retrieval rule is a plain word-matching count, so every result can be checked exactly. This project does not call a model, so “words” stand in for tokens: one word counts as one token. All three tasks are required.

Your evidence file. Keep one plain-text file, chapter2-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print. Where a value is a list of ids or question names, write them comma-separated with no spaces, in the order your code produced them; where a list is empty, write none.

The data and the rules for all tasks

Copy this into your Python file. It is a fictional workplace handbook of three documents, six questions, and for each question the exact wording of its answer (its gold text).

import re

DOCS = {
    "handbook": "Loan laptops are issued by the equipment desk. A loan laptop must be returned within 14 days of the issue date. Staff who need a laptop for longer must ask their manager to approve an extension. Extensions are recorded in the asset register. A laptop that is lost must be reported to the security team on the same day. The security team will then lock the device remotely.",
    "network": "Guest wifi is available in every meeting room. The guest network password changes on the first Monday of each month. Staff devices must use the corporate network instead of the guest network. Printers are only reachable from the corporate network. Visitors who need a printer should ask reception, who will print for them. Remote staff reach the corporate network through the VPN.",
    "expenses": "Expense claims are submitted through the finance portal. Claims below 50 dollars do not need a receipt. Claims of 50 dollars or more must include a receipt image. Receipts older than 60 days will be rejected by the portal. Travel claims must also name the project code. Approved claims are paid in the next weekly payment run.",
}

QUESTIONS = [
    ("q1", "How many days can a loan laptop be kept before it must be returned?", "within 14 days of the issue date"),
    ("q2", "What happens to a lost laptop after it is reported?", "will then lock the device remotely"),
    ("q3", "When does the guest network password change?", "on the first Monday of each month"),
    ("q4", "Do I need a receipt for a 30 dollar expense claim?", "Claims below 50 dollars do not need a receipt"),
    ("q5", "Which claims must name the project code?", "Travel claims must also name the project code"),
    ("q6", "How old can a receipt be before the portal rejects it?", "Receipts older than 60 days will be rejected by the portal"),
]

STOP = set("the a an of to is are in on for be by and or it that this who which what how do i can will must from at with as when".split())

def words(text):
    return re.findall(r"[a-z0-9]+(?:-[a-z0-9]+)*", text.lower())

def contains(chunk_text, gold):
    # True when the gold wording appears whole inside the chunk, ignoring case and punctuation
    return " ".join(words(gold)) in " ".join(words(chunk_text))

def score(question, chunk_text):
    # number of different non-stop words that the question and the chunk share
    return len((set(words(question)) - STOP) & (set(words(chunk_text)) - STOP))

The chunking rule. Split a document’s text into words with text.split(). A chunk is size words joined by single spaces. The first chunk starts at word 0, and each next chunk starts size - overlap words after the previous one. Stop after the first chunk whose end reaches or passes the last word, so the last chunk may be shorter. Chunks are named <document name>-<n>, counting from 1 within each document (handbook-1, handbook-2). The full chunk list visits the documents in the order they appear in DOCS.

A question’s answer is intact when at least one chunk satisfies contains(chunk_text, gold).

The retrieval rule. Rank every chunk by score(question, chunk_text), highest first; when two chunks have the same score, the smaller chunk id (as text) goes first. Drop chunks with a score of 0. Keep the first k. A question is retrieved when at least one kept chunk satisfies contains.

The assembly rule. Go through the kept chunks in rank order, adding each one to the context. Stop at the first chunk that would make the total number of words go over the budget; that chunk and all later ones are left out. A question is assembled when at least one included chunk satisfies contains.

Task 1 — Chunk with two settings

Write two functions.

1. chunk_doc(doc_name, text, size, overlap) — returns a list of (chunk_id, chunk_text) pairs for one document, following the chunking rule.
2. build(size, overlap) — returns the chunks for all three documents, in order.

Then, for setting A (size 12, overlap 0) and setting B (size 30, overlap 10), print:

• the number of chunks;
• how many of the six questions have an intact answer;
• the ids of the questions whose answer is not intact.

Also print the text of chunk expenses-2 under setting A.

Stop there: two settings, six questions. You do not need a model, a vector database or a command-line interface.

Add to your evidence file:

t1_a_chunks: <number of chunks under A>
t1_a_intact: <number of intact answers under A>
t1_a_broken: <question ids not intact under A, or none>
t1_b_chunks: <number of chunks under B>
t1_b_intact: <number of intact answers under B>
t1_b_broken: <question ids not intact under B, or none>
t1_a_expenses2: <the text of chunk expenses-2 under A, exactly>

Output: one Python file with two chunking functions, and seven evidence lines. The last line shows what a chunk that starts in the middle of an answer looks like.

Task 2 — Retrieve with different numbers of chunks

This task stands on its own: it needs the data and rules above and a build function, which you can copy from Task 1 or write again from the chunking rule. Use setting B (size 30, overlap 10) throughout.

Write retrieve(chunks, question, k) — returns the kept (chunk_id, chunk_text) pairs for one question, following the retrieval rule.

Then compute, for k = 1 and k = 2, how many of the six questions are retrieved, and the ids of the questions that are not retrieved. Print the ids of the top two chunks for q5 and q6, and the score of the top-ranked chunk for q1.

Add to your evidence file:

t2_k1_hits: <number of questions retrieved with k = 1>
t2_k1_missed: <ids of questions not retrieved with k = 1, or none>
t2_k2_hits: <number of questions retrieved with k = 2>
t2_q5_top2: <ids of the top two chunks for q5>
t2_q6_top2: <ids of the top two chunks for q6>
t2_q1_top_score: <the score of the top-ranked chunk for q1>

Before you submit, look at your q6 result. Write one sentence for yourself comparing your q6 result at k = 1 with the result at k = 2; it is not scored.

Output: a retrieval function added to your file, and six evidence lines.

Task 3 — Assemble under a budget, then write the chunks file

This task stands on its own: it needs the data and rules above, plus your build and retrieve functions, which you can copy from Tasks 1 and 2 or write again. Use setting B and k = 3.

Write assemble(kept_chunks, budget) — returns the included (chunk_id, chunk_text) pairs, following the assembly rule, with the total number of words.

1. With a budget of 50 words, run all six questions and count how many questions have an intact answer, how many are retrieved and how many are assembled. Print the three counts in that order.
2. Print the ids of the questions that are retrieved but not assembled, the ids of the chunks included for q6, and the total words included for q5.
3. Run the six questions again with a budget of 60 words, and print how many are assembled.
4. Write chunks.tsv: one line per chunk of setting B, in the order build returns them, as the chunk id, a tab character, and the chunk text.

Add to your evidence file:

t3_stage_counts: <intact, retrieved, assembled counts at budget 50, comma-separated>
t3_lost: <ids of questions retrieved but not assembled at budget 50, or none>
t3_q6_included: <ids of the chunks included for q6 at budget 50>
t3_q5_words: <total words included for q5 at budget 50>
t3_budget60_assembled: <number of questions assembled at budget 60>
t3_chunk_rows: <number of lines in chunks.tsv>
t3_chunk_ids: <the ids in chunks.tsv, in file order>

If you completed Chapter 1 with your own passages and gold.tsv, you can run your three functions on them too and see which stage loses your questions; that run is not scored.

Output: an assembly function, a chunks.tsv, and seven evidence lines.

What carries forward

Keep chunks.tsv and chapter2-evidence.txt. The next chapter opens with chunks.tsv: it takes those chunks, makes a generated answer cite the chunk each claim came from, and makes the system refuse when no chunk supports an answer.