Courses / Agentic AI from Scratch (Python)

Session 17 of 40

Better search: top-k, a score threshold, MMR and metadata filters

Notes

How to Tune RAG Search: Top-k, Distance Limit, MMR and Metadata Filters in Python

Agentic AI Foundations · Session 17 — Better search: top-k, a score threshold, MMR and metadata filters

Before you start: what do you need to compare search settings for a RAG helper?

You need the code folder for this session open in your terminal. It holds code/compare_settings.py, code/load_policies.py (which gained two inputs, max_chars and overlap), code/build_index.py (carried over from session 14), code/data/retrieval_set.json with sixteen test questions, data/policies/ with the 5 policies and one prompt file in prompts/. Install the libraries with pip install langchain langchain-openai langchain-chroma langchain-community langchain-text-splitters. Switch on your Python environment first, then set three settings: LLM_BASE_URL (the address of your provider’s OpenAI-compatible endpoint), LLM_API_KEY (your key, kept out of any file you share) and EMBED_MODEL (the name of an embedding model your provider offers). No chat model is called in this session, only the embedding model, and code/compare_settings.py builds its indexes in memory, so it does not need the policy_index/ folder. Recap of session 15: the helper searches an index and puts the TOP = 4 closest chunks in the prompt. Have paper or a notes app ready for the Flashcards and the Model answer.

Summary: which search settings change what a RAG helper retrieves, and how do you test them?

A RAG answer is right only when the search found the right chunks, and a handful of settings decide what it finds. Top-k sets how many chunks come back, chunk size and overlap set how the policies are cut, a distance limit drops chunks that are too far from the question, maximal marginal relevance picks chunks that are close and different from each other, and a metadata filter limits the search to one policy file. Each has a failure side, so you choose by counting hits: change one setting at a time and count how many of the same sixteen questions find their answer words in the returned chunks. In our run (the printed table is in code/README.md, Step 2) the nine settings scored between 14 and 16 hits, and only the distance limit reached 16 of 16, because of the nine it is the one that could return nothing for the question no policy answers.

Key concepts: top-k, chunk size, distance limit, MMR, metadata filter and hit defined

Term Definition
Top-k (k) The number of closest chunks the search returns. The LangChain as_retriever default is 4.
Chunk size (max_chars) The biggest a chunk may be, in characters; a policy section longer than this is cut again.
Overlap (overlap) How many characters a cut chunk repeats from the one before it.
Distance The score the scored by-vector search in code/compare_settings.py is read as: smaller means closer to the question. Check what your store returns by printing it.
Distance limit (max_distance) A setting that drops any chunk whose distance is above the limit; None keeps all.
MMR Maximal marginal relevance: a way of choosing chunks that are close to the question and also different from each other.
fetch_k How many candidates are fetched before MMR picks k of them; the code uses 12.
Metadata filter A condition on chunk metadata, here the policy file in source, that limits which chunks are searched.
Hit A test question whose phrase appears inside one of the returned chunks; for the question with no answer, a hit means the search returned nothing.

Explanations: how do you choose and test search settings for RAG?

Why does the search miss a clause that is right there?

Think of a panel of advisers: if three of the four seats go to people who say the same thing, the fourth view goes unheard. A search can behave the same way. Look at the questions in code/data/retrieval_set.json: “Who signs off my time off?” uses none of the policy’s own words, and the clause it needs says “Your manager approves or declines a request”. The search has to match meaning, return the right chunk among its k, and not be crowded out by closer-looking near-copies. A second picture is a search results page where the first few matter most: what gets shown decides what you learn.

So the question is not “is the search good?” but “how often does the right chunk come back, and which setting changes that?” You answer it with a count.

What is a hit, and how does code/compare_settings.py count it?

An exam marker with an answer key ticks one box per question and adds a total. A hit is the same tick. Each test case has a name, a question, a phrase and a source:

{
  "name": "who signs off leave",
  "question": "Who signs off my time off?",
  "phrase": "Your manager approves or declines a request",
  "source": "leave-policy.md"
}

The case is a hit when its phrase is inside one of the chunks the search returned. The last question, “What is the capital of France?”, has an empty phrase: no policy answers it, so it is a hit only when the search returns nothing. The script embeds all sixteen questions once, in one call, and every setting reuses those numbers, so comparing nine settings costs little. A count over sixteen questions is a rough signal, not a proof, but it is far better than judging by one example.

What is top-k and how do you set it?

A shortlist for a job shows the idea: too short and the best person is missing, too long and the panel is overwhelmed. Top-k is the number of closest chunks returned. LangChain’s reference for as_retriever lists k as the amount of documents to return, with a default of 4, which is what session 15 used.

In our run, k=1 scored 14 of 16 and missed “lost receipt”, while k=2 scored 15 of 16 and found it: one chunk is too few when the answer sits in the second-closest chunk. Going from k=2 to k=4 found nothing new here. A bigger k costs more tokens in each prompt, so choose the smallest k that does not lose hits.

Do chunk size and overlap change what the search finds?

A binder cut into pages, versus a binder cut into paragraphs: the page holds more, but a single question may match it less well. load_chunks(max_chars, overlap) in code/load_policies.py cuts each policy at its headings and then cuts any section longer than max_chars, so the two sizes are inputs:

def load_chunks(max_chars=MAX_CHARS, overlap=OVERLAP):
    ...
    by_size = RecursiveCharacterTextSplitter(
        chunk_size=max_chars, chunk_overlap=overlap)
    return by_size.split_documents(sections)

code/compare_settings.py tried 600 characters (34 chunks), 250 characters (44 chunks), 250 with an overlap of 60 (45 chunks) and 1200 characters (33 chunks). Each pair needs its own index with its own collection_name, or two sizes would share one set of chunks. In our run, small chunks and small chunks with overlap scored 15 of 16, the same as 600, and big chunks scored 14 of 16 and lost “same week away”. A chunk that holds several clauses gets a blurred meaning, so a question about one clause can match it less well.

What is a distance limit, and which of our settings can return nothing?

A pass mark on an exam works like a limit: below it, no certificate, however few people passed. A bouncer with a guest list is the second picture: a name close to the list is not enough. In code/compare_settings.py (step 6 in its comments), the scored by-vector search similarity_search_by_vector_with_relevance_scores returns each chunk with a number that the code treats as a distance, where smaller means closer, and the setting max_distance drops chunks above the limit:

scored = index.similarity_search_by_vector_with_relevance_scores(
    vector, k=setting["k"], filter=where)
limit = setting.get("max_distance")
return [chunk.page_content for chunk, distance in scored
        if limit is None or distance <= limit]

Do not mix this number up with LangChain’s relevance score, which its reference says runs from 0 (dissimilar) to 1 (most similar), where higher is closer. LangChain’s retriever option score_threshold is a minimum on that 0-to-1 scale, and this code does not use it. Which scale a store’s scored search returns can differ by store and method, so print a few values for a close and a far chunk before you write a limit.

Failure first: the question about France is answered by no policy, yet an ordinary search still returns its four closest chunks, however far away they are. In our run (code/README.md, Step 2) the eight settings without a distance limit all returned chunks for it, so none counted it as a hit. Only max_distance 0.9 returned nothing for it, and it scored 16 of 16. Among the nine settings it is the only one built to return nothing; a metadata filter can also return nothing, but only when no chunk’s metadata match. That matters for session 16’s helper: from chunks that are not about the question, it has to say “I don’t know” itself. The risk is strictness: distances depend on the embedding model, so a limit of 0.9 that works on ours may return nothing for every question on yours. Print the distances for a good match and a bad one, then pick a limit between them.

What is MMR and why does it give more varied results?

A tasting menu has one dish from each course, not four desserts. MMR works like that. LangChain’s vector store reference says maximal marginal relevance optimizes for similarity to the query and diversity among the selected documents. In the code you fetch more candidates than you keep (fetch_k=12) and MMR picks k of them with max_marginal_relevance_search_by_vector.

MMR helps when the closest chunks repeat each other. In our run it did not help: k=4, mmr scored 14 of 16 and lost “lost receipt”, where one clause was all the question needed and variety pushed it out. A setting is not better because it sounds clever; the count decides.

Searching one drawer rather than the whole cabinet is quicker and clearer. A metadata filter limits the search to chunks whose metadata match. In the code the setting by_source filters on the policy file that the test case names:

where = ({"source": case["source"]}
         if setting.get("by_source") and case["source"] else None)

The source label from session 12 is what makes this possible. In our run k=4, by source matched plain k=4 at 15 of 16, because the closest chunks were already in the right file. It can only be used when you already know which policy holds the answer, and a wrong filter hides the answer: the right chunk is never a candidate.

How do you read the whole table and know a change helped?

Run python code/compare_settings.py from the folder that holds code/. Ours printed (the same table is in code/README.md, Step 2):

setting                    chunks  hits
k=1                            34  14/16
k=2                            34  15/16
k=4 (session 15)               34  15/16
k=4, small chunks              44  15/16
k=4, small + overlap           45  15/16
k=4, big chunks                33  14/16
k=4, max_distance 0.9          34  16/16
k=4, mmr                       34  14/16
k=4, by source                 34  15/16

Each real line also ends with missed: and the names of the questions missed; the France question is missed by every setting except the distance limit. Your numbers may differ a little with another embedding model. If every setting shows the same hits, the questions are too easy for your model: add harder ones to code/data/retrieval_set.json. Change one setting at a time and count: if you change three at once and it “seems better”, you cannot say which helped.

Three situations where the search misleads

  1. “What is the capital of France?” still returns four chunks. In our run the eight settings without a distance limit returned k chunks however far away, so the helper has to say “I don’t know” from chunks that are not about the question.
  2. max_distance 0.9 returns nothing for every question on a different embedding model. Distances depend on the model, so print them and pick a limit between a good match and a bad one.
  3. A filter on the wrong policy file. The search only looks in that file, so the right chunk is never a candidate.
What you learned The design decision Where it shows
One chunk is too few when the answer is second closest Choose the smallest k that loses no hits k=1 against k=2
A search returns chunks however far away Add a distance limit set from printed distances max_distance
Variety is not always better Count hits before keeping MMR mmr
A filter needs a known policy Use by_source only when the topic is known by_source

Basic assignment: add a setting with max_distance 0.5, then 0.7, and watch the hit count and what the first question returns. Too tight a limit throws away the right chunk. Then print the distance of the closest chunk for the France question and for the leave question and pick your own limit.

Project step: compare, then choose. Run python code/compare_settings.py and write down your own hit counts. Set MAX_CHARS in code/load_policies.py to the chunk size you prefer and run python code/build_index.py: that is the index later sessions search. Add three questions of your own to code/data/retrieval_set.json that you think the search will miss. The build prompt is code/prompts/1-write-compare-settings.md. Stop when the script prints nine lines and you have your counts.

Practice: explain and diagnose. First, answer the Model answer question below aloud in three beats before reading it. Second, for each situation, say which setting you would look at:

  1. A question about the capital of France still returns four chunks.
  2. The right clause is the second closest and a single chunk is returned.
  3. Every question returns nothing after you add a limit.

Flashcards: can you answer these search tuning questions aloud?

Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.

1. What does top-k control and what is the LangChain default? The number of closest chunks the search returns. The `as_retriever` default is 4. Too few misses the answer and too many costs more tokens in each prompt.
2. What is a hit? A test question whose `phrase` appears inside one of the returned chunks. For the one question with no answer, a hit means the search returned nothing.
3. Why did `k=2` find a question that `k=1` missed? The answer sat in the second-closest chunk. One chunk is too few then, and a second one brings it in.
4. What do `max_chars` and `overlap` set? `max_chars` is the biggest a chunk may be, and `overlap` is how many characters a cut chunk repeats from the one before it. Each pair needs its own index.
5. What does the distance mean in this code? How far a chunk is from the question: smaller means closer. It is not LangChain's 0-to-1 relevance score, where higher means closer; print the values from your own store to be sure which you have.
6. Which of our nine settings could return nothing, and why? The distance limit: it drops chunks that are too far away. The other eight return `k` chunks however far away they are, so the France question got chunks from them and was not a hit.
7. What is the risk of a distance limit? Distances depend on the embedding model, so a limit that works on one model may return nothing for every question on another. Print the distances before choosing.
8. What is MMR and what did it do in our run? Maximal marginal relevance picks chunks that are close to the question and different from each other. In our run it scored 14 of 16 and lost a hit, because one clause was all the question needed.
9. What does a metadata filter do, and what is its danger? It limits the search to chunks whose metadata match, such as one policy file. A wrong filter hides the answer, because the right chunk is never a candidate.
10. Why change one setting at a time? If you change three and the count looks better, you cannot tell which one helped, or whether one helped and another hurt.

Model answer: how do you explain tuning the search in a RAG system?

The core question this session answers: “A colleague asks: the helper sometimes misses the right clause. What can we change in the search, and how do we know it helped?”

Three beats, in this order: name the dials · change one and count hits · choose by the numbers. Skip any of these three and the answer is incomplete.

  1. Name the dials: k, chunk size and overlap, a distance limit, MMR and a metadata filter.
  2. Change one and count hits: run the same sixteen questions through each setting and count how many returned chunks contain the answer words.
  3. Choose by the numbers: keep the smallest k that loses no hits, and a distance limit, set from printed distances, if the helper must be able to find nothing.

The full answer, said aloud:

(Name the dials) “The search has a few dials: how many chunks come back, how the policies are cut, a distance limit that drops far chunks, MMR for variety, and a filter for one policy file.

(Change one and count hits) I change one dial at a time and count, over the same sixteen questions, how many get their answer words in the returned chunks. If I changed three at once I could not say which helped.

(Choose by the numbers) I keep the smallest k that loses no hits. If the helper must be able to find nothing, I add a distance limit, set from the distances I printed. I keep MMR or a filter only when the count rises.“

Missing a beat is a failure state. The follow-up question lands where you went quiet: skip the last beat and the next question is how you know it helped. Practise until the three come out without prompting.

Why this matters: where does search tuning matter in real systems?

In a debugging session: when a document assistant gives a wrong answer, the search is one of the places to look first, as session 15 showed with the retrieved list. Knowing the dials means you can say what to change rather than “try a better model”.

On the job: a hit count on a small, fixed set of questions is a cheap check on a retrieval feature. It can show that a setting change quietly made things worse.

In the rest of this course: session 18 adds keyword and hybrid search, a reranker and query rewrites, which you judge with the same kind of hit count, and the evaluation sessions grow the test set into a proper measure.

Assignment. Add a setting with max_distance 0.5, then 0.7, and say what happens to the hit count. The brief for the project work, compare then choose, is the project file. Keep your index: later sessions search it.

What’s next. Session 18 covers keyword search, hybrid search, a reranker and rewriting the question: more ways to find the right clause when meaning search alone misses.

Want help building this with someone?

Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.

Ask about mentor-led training

TechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.