Courses / Agentic AI from Scratch (Python)
Session 15 of 40
How RAG works: retrieve, add to the prompt, generate
Notes
How to Build RAG in Python: Retrieve, Add to the Prompt, Generate
Agentic AI Foundations · Session 15 — How RAG works: retrieve, add to the prompt, generate
Before you start: what do you need to build retrieval-augmented generation in Python?
You need the code folder for this session, chapters/15-rag-helper/, open in your terminal. It holds helpdesk_rag.py, run_test_set.py, build_index.py and load_policies.py (carried over from sessions 14 and 12), model_prompts/rag.md, data/test_set.json with eleven cases, data/policies/ with the 5 policies, and two prompt files in prompts/. Install the libraries with pip install langchain langchain-openai langchain-chroma langchain-community langchain-text-splitters. You need the settings from session 1 (LLM_BASE_URL, LLM_API_KEY, LLM_MODEL) and the embedding setting from session 13 (EMBED_MODEL). EMBED_MODEL must be the model that builds the index, so set it once in your terminal rather than in two places. The folder policy_index/ is not in the lesson files: you build it in this session. Have paper or a notes app ready for the Flashcards and the Model answer.
Summary: what is retrieval-augmented generation and how does it answer from your documents?
Retrieval-augmented generation, or RAG, finds the pieces of your own documents that relate to a question, adds them to the prompt and has the model write the answer from them. It exists because a model cannot take in a whole library at once and its training knowledge is frozen at a point in time. The helper’s version has three steps: search the Chroma index for the closest chunks, write them into one prompt with their source files, and call the model, which returns a typed reply that now includes a source field. The answer depends on the chunks it was given, so a wrong answer has three places to look: the search missed the right chunk, the prompt let the model answer badly, or the index is out of date. Printing the retrieved chunks beside the answer tells you which one you have, and the test set now checks the cited source as well as the answer.
Key concepts: RAG, retriever, context and source terms defined
| Term | Definition |
|---|---|
| RAG | Retrieval-augmented generation: find the relevant pieces of your own documents, add them to the prompt, and have the model answer from them. |
| Retriever | An object with one job: give it a question, get back the closest chunks. Here index.as_retriever(search_kwargs={"k": TOP}). |
TOP |
The setting for how many chunks the search hands to the model; 4 in this session. |
| Context | The text of the retrieved chunks, written into the prompt’s {context} blank, each with its source and heading. |
rag.md |
The prompt file the helper reads, model_prompts/rag.md, with a {context} blank and the rules for a missing answer. |
source |
The file name of the chunk used, such as learning-policy.md. It is chunk metadata from session 12 and now a field of the typed reply. |
retrieved |
The list ask_helper returns beside the answer: the source and heading of each chunk the search found. |
ask_helper(question) |
The function that runs the three RAG steps and returns the reply as a dictionary, or None if anything fails. |
Explanations: how do you build RAG step by step?
Why not paste every policy into the prompt?
Think of a student who is handed every textbook at the exam desk. The answer is in there somewhere, but finding it is the hard part. In session 9 the whole of every policy went into every prompt. With 5 policies, and more later, that stops being practical. LangChain’s retrieval guide names the two limits behind it: a model has finite context, so it cannot ingest entire corpora at once, and static knowledge, since its training data is frozen at a point in time. Retrieval addresses them by fetching the relevant knowledge when the question is asked.
A colleague who is handed the page they need, and not the whole binder, is the picture to keep. In the helper, only four chunks go into the prompt.
What is RAG, and what are its three steps?
An open-book exam is the first picture: the student looks up the right page, then writes the answer. A librarian who fetches the pages first is the second: you write your summary with the pages in front of you. RAG has the same shape:
Question --> 1. Retrieve: search the index for the closest chunks
|
2. Add to the prompt: instructions + chunks
(each with its source) + question
|
3. Generate: the model writes the typed reply
|
answer, section, source, needs_human
LangChain calls this plain design 2-step RAG: the retrieval step always runs before the generation step, which makes it straightforward and predictable, and it suits cases such as FAQs and documentation bots. A practical benefit follows: a changed policy means a changed file and a rebuilt index, not a retrained model.
How does the code retrieve chunks and put them in the prompt?
Step 1 is the retriever. It opens the folder from session 14 and wraps the search:
index = Chroma(persist_directory=str(INDEX_FOLDER),
embedding_function=embeddings)
retriever = index.as_retriever(search_kwargs={"k": TOP})
Give the retriever a question and it returns the TOP closest chunks. If the folder is missing, the script stops with “No index yet. Run: python build_index.py”.
Step 2 is the prompt. model_prompts/rag.md is a template with a {context} blank. It tells the model to answer only from the pieces, to say so when they do not answer the question, and to copy the source of the piece it used. It also lists the cases where needs_human must be true and section and source must be “none”: when the pieces do not answer the question, when the person asks what a named policy says and none of the pieces comes from a policy with that name, and when the question is not about help desk topics. That is the rule-for-a-missing-case idea from session 8, now for retrieved text. In ask_helper, the chunks are written into one text, each with its source and heading above it:
chunks = retriever.invoke(question)
context = "\n\n".join(
f"[{number}] source: {chunk.metadata['source']} | "
f"{chunk.metadata.get('heading')}\n{chunk.page_content}"
for number, chunk in enumerate(chunks, start=1))
The source is the metadata you kept in session 12; that is what lets the helper cite a file. Compare with session 9, where the whole of every policy went into every prompt. Now only four chunks go in. That is cheaper, and it still works when the policies grow too large to paste.
How does the helper return its answer with a source?
The typed reply from session 9 gained one field:
class HelpDeskAnswer(BaseModel):
answer: str = Field(description="...")
section: str = Field(description="...")
source: str = Field(description="...")
needs_human: bool = Field(description="...")
source is the file name of the piece used, or “none”. A footnote keeps a claim and its source together; this field does that for each answer. Because it is a value, not a sentence, your code can check it, and run_test_set.py does. ask_helper also adds a retrieved list to the reply, so a script can see which chunks the search found. Build the index with python build_index.py (it prints Saved 34 chunks to policy_index/), then run python helpdesk_rag.py. Ours printed:
Question: How much is my yearly learning budget?
retrieved:
- learning-policy.md / 2. Learning budget
- learning-policy.md / 3. How to request it
- learning-policy.md / 4. Learning days
- learning-policy.md / 1. Purpose
answer: Every employee has a learning budget of 600 for each leave year.
section: Learning Policy 2.1
source: learning-policy.md
needs_human: False
Read the retrieved list first. It is the evidence: those four chunks are all the model saw of the 5 policies. The answer comes from the first one. Ask your own question: python helpdesk_rag.py "Can I work from another country?".
What goes wrong in RAG: the right chunk not found, or found and answered badly?
A reader who is handed the wrong page writes the wrong answer, however carefully they write. From outside, three failures look alike, because each produces a wrong answer. Printing the retrieved list tells them apart.
| What you see | Where to look | What to fix |
|---|---|---|
The right chunk is not in the retrieved list |
The search | TOP, or the chunk size in load_policies.py |
| The right chunk is there, and the answer is wrong | The prompt | rag.md |
| A correct reading of an old figure | The index | Run build_index.py again |
Research on long inputs, titled “Lost in the Middle”, found that model performance is often highest when the relevant information is at the beginning or end of the input and degrades significantly when it is in the middle of a long context. That is one reason to hand the model a few chunks rather than many.
How does the test set check the cited source?
run_test_set.py is session 8’s runner with one more check. A case may name the source file the answer must cite:
elif "source" in case and reply["source"] != case["source"]:
ok = False
why = (f"source was '{reply['source']}', "
f"expected '{case['source']}'")
data/test_set.json has eleven cases: the eight from session 9, four of them now also naming a source, and three new ones about remote work and learning. A case fails if the reply needs a person when it should not (or the other way round), if the answer lacks the expected word, or if the source is not the expected file. Checking the cited file is a stronger test than checking the answer text alone, because an answer can contain the right word and still come from the wrong policy. Run python run_test_set.py. Ours printed:
PASS - carry-over days -
PASS - who approves leave -
PASS - lost laptop -
PASS - claiming an expense -
PASS - remote days -
PASS - learning budget -
PASS - working abroad -
PASS - section that does not exist -
PASS - policy that does not exist -
PASS - sick leave allowance -
PASS - off-topic question -
Passed 11 of 11
Run it more than once, as in session 8. When a case fails, look at its retrieved chunks with python helpdesk_rag.py "<the question>": if the right chunk is not in the list, fix the search; if it is there and the answer is still wrong, fix rag.md.
What are three situations where RAG answers go wrong, and what does each mean for the design?
- A correct-looking answer comes back with a wrong or
nonesource. The chunk the model used may not be the one you expect, so read theretrievedlist. - Answers about a policy you just edited are old. The index is a copy, and it was not rebuilt with
build_index.py. - The helper answers “what does the travel policy say?” from a chunk that only mentions travel. The prompt must say that a named policy absent from the pieces needs a person, even if another piece mentions the topic.
| What you learned | The design decision | Where it shows |
|---|---|---|
| A model cannot take in every policy | Retrieve a few chunks per question | retriever, TOP = 4 |
| An answer needs a checkable source | Add a source field to the typed reply |
HelpDeskAnswer |
| Three places a wrong answer can start | Print the retrieved list beside the answer |
ask_helper |
| The index is a copy | Rebuild it when a policy changes | build_index.py |
Basic assignment: set TOP = 1 and run the test set. Which cases break first, and why? Then edit data/policies/learning-policy.md (change the 600 budget to 900), run the test set without rebuilding the index, then rebuild and run again. The helper answers from the old text until you rebuild. Why?
Project step: answer from retrieved chunks. Run python build_index.py, python helpdesk_rag.py and python run_test_set.py in order and write down your pass count. Add a policy file of your own, rebuild, and add a test case with its source. The build prompts are prompts/1-write-helpdesk-rag.md and prompts/2-write-run-test-set.md. Stop when the test set runs and you have a count.
Practice: explain and diagnose. First, answer the Model answer question below aloud in three beats before reading it. Second, for each situation, say whether to look at the search, the prompt or the index:
- The
retrievedlist holds only chunks about learning, and the question was about remote work. - The right chunk is listed, but the answer contradicts it.
- The helper still says 600 after you edited the file to 900.
Flashcards: can you answer these RAG questions aloud?
Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.
1. What is RAG, in one sentence?
Finding the pieces of your own documents that relate to a question, adding them to the prompt, and having the model write the answer from them.2. What are the three steps of RAG in our helper?
Retrieve the closest chunks from the Chroma index, add them to the prompt with their sources, and generate the typed reply.3. Why not put every policy into every prompt?
A model has finite context, so it cannot take in a whole library at once, and its training knowledge is frozen at a point in time. Retrieval fetches the relevant pieces when the question is asked.4. What is `TOP`?
How many closest chunks the retriever hands to the model; it is 4 in this session.5. What does the `{context}` blank in `rag.md` hold?
The retrieved chunks, each written with its source and heading above it, so the model can say where its answer came from.6. What new field did the typed reply gain, and why does it matter?
`source`, the file name of the piece used. It is a value your code can check, not a sentence to read, and `run_test_set.py` uses it.7. What does the `retrieved` list show?
The source and heading of each chunk the search found: the evidence, since those chunks are all the model saw of the policies.8. How do you tell a search failure from an answer failure?
Read the `retrieved` list. If the right chunk is missing, fix the search (`TOP`, chunk size). If it is there and the answer is still wrong, fix `rag.md`.9. Why does the helper answer from old text after you edit a policy?
The index is a copy of the files, so it keeps the old text until you run `build_index.py` again.10. Why is checking the cited source a stronger test?
An answer can contain the expected word and still come from the wrong policy. Checking that `source` is the expected file catches that.Model answer: how do you explain how an AI assistant answers from company documents?
The core question this session answers: “A colleague asks: how does the helper answer from our policies instead of from a whole file, and where do I look when an answer is wrong?”
Three beats, in this order: retrieve · add to the prompt · generate and cite. Skip any of these three and the answer is incomplete.
- Retrieve: the retriever searches the Chroma index and returns the
TOPclosest chunks. - Add to the prompt: each chunk is written in with its source and heading, so the model answers from those pieces.
- Generate and cite: the model returns the typed reply, including the
sourcefile; when an answer is wrong, read theretrievedlist first to tell a search failure from an answer failure.
The full answer, said aloud:
(Retrieve) “The question is turned into numbers and searched against the saved index, and the retriever returns the four closest chunks, not the whole policy.
(Add to the prompt) Those chunks go into the prompt’s context blank, each with its source file and heading, and the prompt tells the model to answer only from them and to say so when they do not answer.
(Generate and cite) The model writes the typed reply, with a
sourcefield naming the file it used. If an answer is wrong, I print the retrieved list first: if the right chunk is missing I fix the search, if it is there I fix the prompt, and if the figure is old I rebuild the index.“
Missing a beat is a failure state. The follow-up question lands where you went quiet: skip the last beat and the next question is how you would find out why an answer was wrong. Practise until the three come out without prompting.
Why this matters: where does RAG matter in real systems?
When explaining RAG to a manager: the plain answer is that the assistant looks up the right pages first and then writes from them, which is why a changed policy needs a rebuilt index, not a retrained model.
When a reviewer asks where an answer came from: the source field and the retrieved list give a checkable answer instead of a guess.
In the rest of this course: session 16 covers citing sources, saying “I don’t know” and checking an answer is grounded; session 17 tunes how many chunks to retrieve; and the evaluation sessions score the test set with a model that grades the answers.
After this unit: what is the assignment and what comes next?
Assignment. Set TOP = 1, run the test set, and say which cases break first and why. The brief for the project work is the project file. Keep your index and your test set: later sessions re-run them.
What’s next. Session 16 covers citing sources, saying “I don’t know”, and checking that an answer is grounded in the retrieved chunks.
Want help building this with someone?
Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.
Ask about mentor-led trainingTechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.