Courses / Agentic AI from Scratch (Python)

Session 19 of 40

Lab and Milestone 1: answers from documents, with sources

Notes

How to Test and Debug a RAG Chatbot Before Showing It: The Milestone 1 Lab

Agentic AI Foundations · Session 19 — Lab and Milestone 1: answers from documents, with sources

Before you start: what do you need to test your copilot as a whole?

You need the code folder for this session open in your terminal. It holds search.py (the whole search in one file), helpdesk_rag.py (the copilot), run_test_set.py, build_index.py and load_policies.py (carried over from sessions 14 and 12), two prompt files in model_prompts/ (rag.md and rerank.md), data/test_set.json with eighteen cases, data/policies/ with the 5 policies and two prompt files in prompts/. Install the libraries with pip install langchain langchain-openai langchain-chroma langchain-community langchain-text-splitters, and set LLM_BASE_URL, LLM_API_KEY, LLM_MODEL and EMBED_MODEL as in earlier sessions. Build the index with python build_index.py, which prints Saved 34 chunks to policy_index/. Recap of earlier sessions: session 16 gave the helper a quote, a citation check and the “I don’t know” rule, and session 18 gave it hybrid search and a rerank. Have paper or a notes app ready for the Flashcards and the Model answer.

Summary: what does Milestone 1 require and how do you read a failing RAG test?

A milestone is a checkpoint made of checks you can run. Milestone 1 is the finished help desk copilot: search.py finds four chunks with meaning search, word search, fusion and a rerank, and ask_helper in helpdesk_rag.py answers with a source and a quote, checks that the quote is inside a retrieved chunk, and says “I don’t know” when nothing supports an answer. The test set behind it has eighteen questions that mix answerable questions, exact clause labels, questions in other words and questions the policies do not answer, because a set of easy questions measures only the easy part. When a case fails, you read the retrieved chunks first and name the failing part, the search, the prompt or the citation, before you change anything. On our run the test set passed 18 of 18, and the number you can repeat matters more than one lucky run.

Key concepts: milestone, test set, find_chunks, ask_helper and failure triage defined

Term Definition
Milestone A project checkpoint defined by checks you can run, rather than by a general impression.
find_chunks(question) The one function in search.py that the copilot calls: it runs meaning search and word search, fuses the lists, reranks the candidates and returns the chunks the model reads.
FETCH, CANDIDATES, TOP Three numbers in search.py: how many chunks each search gives the fusion (6), how many fused chunks go to the rerank (6) and how many the model reads (4).
ask_helper(question) The copilot’s function: gets chunks from find_chunks, returns a typed reply with a source and a quote, and returns None if the call fails.
Quote check quote_is_real: the quote must be inside a retrieved chunk of the cited source, or the answer is replaced by “I don’t know”.
Test set A fixed list of questions with what a good reply must contain, run again after each change.
Pass count The line Passed X of Y that run_test_set.py prints.
Failure triage Naming which part failed, the search, the prompt or the citation, before changing anything.

Explanations: how do you assemble, test and debug the whole copilot?

Why is “it seems to work” not enough?

Think of a driving test with a checklist: the examiner ticks specific manoeuvres, not an overall impression. A building inspection works the same way, item by item. The failure looks like this: a team says the assistant “seems to work”, a manager asks which questions were tried, and nobody can show them. OpenAI’s evaluation guide lists “vibe-based evals”, using “it seems like it’s working” as the evaluation strategy, as an anti-pattern. The cause is that no fixed set of questions was run, so there is no evidence and no way to see whether a change helped or hurt.

What is the finished copilot made of?

A kitchen has a prep station and a pass: one place gets the ingredients ready, another plates the dish. The copilot is split the same way. search.py is the prep station and helpdesk_rag.py is the pass.

 question
    |
 find_chunks        search.py: meaning search + word search,
    |               fused, reranked, TOP = 4 chunks (session 18)
 ask_helper         helpdesk_rag.py: chunks into the prompt,
    |               typed reply with a quote (session 16)
 quote_is_real?     no: replaced by "I don't know"
    |
 run_test_set.py    Passed X of 18

The search is one function:

def find_chunks(question):
    candidates = fuse([index.similarity_search(question, k=FETCH),
                       keyword_search(question)])
    return rerank(question, candidates)

search.py holds sessions 17 and 18 in one place: the word search, the fusion and the rerank, with the three numbers FETCH = 6, CANDIDATES = 6 and TOP = 4 to tune. The rerank costs one chat call per question, and the answer is a second call. In helpdesk_rag.py there is one change from session 16:

def ask_helper(question):
    try:
        chunks = find_chunks(question)      # new in session 19
        context = ...                       # chunks, with sources
        reply = chain.invoke({"question": question,
                              "context": context}).model_dump()
        # quote_is_real and the "I don't know" rule: session 16
        ...
        return reply
    except Exception as error:
        print("The chain failed:", error)
        return None

The chunks now come from find_chunks, not from one meaning search. ask_helper keeps its shape, because session 21 calls it as a tool.

What does the search find, and what does the copilot say?

Run python search.py. Ours printed:

Question: What does clause 4.2 of the leave policy say?
  1. leave-policy.md / 4. Annual leave
  2. learning-policy.md / 4. Learning days
  3. leave-policy.md / 4. Annual leave
  4. leave-policy.md / 2. Leave year

These four chunks are all the model will see. The first is the one we want; the second is a weaker match that still reaches the model. Then run python helpdesk_rag.py. Ours printed (the quote line and the retrieved list are trimmed):

Question: What does clause 4.2 of the leave policy say?
retrieved:
  - leave-policy.md / 4. Annual leave
  ...
answer: Up to 5 unused annual-leave days carry over into the next leave year. Carried-over days must be used by 31 March; days not used by then are lost.
section: Leave Policy 4.2
source: leave-policy.md
needs_human: False

Read retrieved first, as in session 15. Then ask your own question: python helpdesk_rag.py "Can I work from another country?".

What makes a good test set?

A driving test held only in an empty car park measures the easy part and little else. OpenAI’s evaluation guide says to ensure test data includes typical cases, edge cases and adversarial cases, and says evaluation is a continuous process. data/test_set.json has eighteen cases: session 16’s thirteen, plus two that name a clause, one in other words, one about learning days and one clause that does not exist.

Kind of case How many in data/test_set.json
Expects an answer and a source file 11, including two that name a clause and one in other words
Expects “I don’t know”, needs_human true and source none 7, including a clause that does not exist

Questions with no answer test the honest reply, and a clause label tests the search. Run python run_test_set.py. Ours printed (trimmed):

PASS - carry-over days -
...
PASS - clause label -
PASS - clause label, another policy -
PASS - different words -
PASS - learning days -
PASS - clause that does not exist -

Passed 18 of 18

Run it more than once. A model’s answers can change between runs: OpenAI’s guide says models sometimes produce different output from the same input, so a number you can repeat is worth more than one lucky run.

When a case fails, which part broke?

A garage runs the same few checks before it opens the engine: fuel, spark, air. The copilot has three places to look, and reading retrieved tells you which.

What you see The part that failed Where to look
The right chunk is not in retrieved The search search.py, FETCH, TOP, load_policies.py
The right chunk is there and the answer is wrong The prompt model_prompts/rag.md
The answer is right but the test fails on source The citation or the check Print the reply: the model cited another file, or the quote check replaced the answer

Three situations at sign-off

  1. A pass count of 17 of 18 on one run and 18 of 18 on the next (an example, not our output). A model’s answers can change between runs, so repeat the run and read the case that failed.
  2. Answers about a policy you just edited are old. The index is a copy, and it was not rebuilt with python build_index.py.
  3. A case that names a clause fails on source. The model cited another file, or the quote check replaced the answer; print the reply to see which.
What you learned The design decision Where it shows
The copilot needs one search entry point Put the search behind find_chunks search.py
A run can differ from the last Repeat the test set and trust a repeatable count Passed X of 18
A failure has three places to start Read retrieved first, then the prompt, then the citation the table above
The index is a copy Rebuild it after each policy edit build_index.py

Ready to show means a pass count you can repeat, and a failing case you understand: you can name the part that broke.

Basic assignment: set TOP = 1 in search.py and run the test set. Which cases break first, and why? Then replace the last line of find_chunks with return candidates[:TOP] to skip the rerank, and see whether the pass count changes.

Project step: run the finished copilot. Run python build_index.py, python search.py, python helpdesk_rag.py and python run_test_set.py in order and write down your pass count. Add three cases to data/test_set.json: one the policies answer, one they do not and one that names a clause. The build prompts are prompts/1-write-search.md and prompts/2-write-helpdesk-rag.md. Stop when the test set runs and you have a count.

Practice: explain and diagnose. First, answer the Model answer question below aloud in four beats before reading it. Second, for each situation, say which part you would look at:

  1. The retrieved list holds only learning chunks, and the question was about remote work.
  2. The right chunk is listed, but the answer contradicts it.
  3. The helper still gives an old figure after you edited a policy.

Flashcards: can you answer these copilot testing questions aloud?

Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.

1. What does `find_chunks` do? It is the one function the copilot calls for its search: meaning search and word search, fused, then reranked, returning the chunks the model reads.
2. What are `FETCH`, `CANDIDATES` and `TOP`? The numbers to tune: how many chunks each search gives the fusion (6), how many fused chunks go to the rerank (6) and how many the model finally reads (4).
3. What changed in `ask_helper` from session 16? Only where the chunks come from: they now come from `find_chunks` (hybrid search and rerank) instead of one meaning search. Its shape stays the same, because session 21 calls it as a tool.
4. What does the quote check do? `quote_is_real` tests that the quote is inside a retrieved chunk of the cited source. If it is not, the answer is replaced by an "I don't know" reply.
5. Why is "it seems to work" not enough? No fixed set of questions was run, so there is no evidence and no way to tell whether a change helped. OpenAI's guide lists this as an anti-pattern called vibe-based evals.
6. What should a test set mix? Typical cases, edge cases and questions the policies do not answer, plus exact clause labels and questions in other words. A set of easy questions measures only the easy part.
7. Why run the test set more than once? A model's answers can change between runs, so a count you can repeat is worth more than one lucky run.
8. A case fails and the right chunk is not in `retrieved`. Which part broke? The search. Look at `search.py`, `FETCH`, `TOP` and `load_policies.py`.
9. The right chunk is there and the answer is wrong. Which part broke? The prompt. Look at `model_prompts/rag.md`.
10. The answer is right but the test fails on `source`. What do you do? Print the reply. Either the model cited another file, or the quote check replaced the answer.

Model answer: how do you explain whether the copilot is ready to show?

The core question this session answers: “A colleague asks: is the copilot ready to show, and if a question fails, how do you find out why?”

Four beats, in this order: one search function · a cited, checked answer · a test set with a count · find the failing part. Skip any of these four and the answer is incomplete.

  1. One search function: find_chunks joins meaning search, word search, fusion and rerank.
  2. A cited, checked answer: ask_helper returns a source and a quote, and the quote must be inside a retrieved chunk, or the answer becomes “I don’t know”.
  3. A test set with a count: 18 questions, including ones the policies do not answer, run with run_test_set.py until the pass count repeats.
  4. Find the failing part: read retrieved first, then the prompt, then the cited source.

The full answer, said aloud:

(One search function) “The copilot gets its chunks from one function, find_chunks, which runs meaning search and word search, fuses them and reranks to four chunks.

(A cited, checked answer) The copilot answers with a source and a quote, and my code checks the quote is inside a retrieved chunk, or the answer becomes I don’t know.

(A test set with a count) I run a test set of eighteen questions, including ones the policies cannot answer, until the pass count repeats, because the model’s answers can change between runs.

(Find the failing part) When a case fails I read the retrieved list first: if the right chunk is missing it is the search, if it is there and the answer is wrong it is the prompt, and if only the source fails I print the reply to see the citation.“

Missing a beat is a failure state. The follow-up question lands where you went quiet: skip the last beat and the next question is how you would find out why a case failed. Practise until the four come out without prompting.

Why this matters: where does a tested RAG copilot matter at work?

In a sign-off meeting: a manager who asks “is it ready?” is asking for evidence. A pass count you can repeat and a failing case you can explain are answers you can show.

On a support team: when a bad answer is reported, the three-way check, search, prompt or citation, turns an argument about the model into a look at one list.

In the rest of this course: session 20 starts the agents module, and session 21 calls ask_helper as a tool, which is why its shape must not change. The evaluation sessions turn this test set into a proper measure.

After this unit: what to build after the finished copilot

Assignment. Set TOP = 1, run the test set, and say which cases break first and why. The brief for the project work, run the finished copilot, is the project file. Keep your index and your test set: later sessions re-run them.

What’s next. Session 20 introduces what an agent is: the reason, act, observe loop.

Want help building this with someone?

Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.

Ask about mentor-led training

TechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.