Courses / Agentic AI from Scratch (Python)
Session 18 of 40
Hybrid search, reranking and rewriting the question
Notes
How to Improve RAG Search: Hybrid Search, Reranking and Query Rewriting in Python
Agentic AI Foundations · Session 18 — Hybrid search, reranking and query rewriting
Before you start: what do you need to add hybrid search, a reranker and a query rewrite?
You need the code folder for this session open in your terminal. It holds code/hybrid_search.py, code/rerank.py, code/rewrite_query.py, code/build_index.py and code/load_policies.py (carried over from sessions 14 and 12), two prompt files in code/model_prompts/ (rerank.md and rewrite.md), code/data/search_set.json with fifteen questions, data/policies/ with the 5 policies and three prompt files in prompts/. Install the libraries with pip install langchain langchain-openai langchain-chroma langchain-community langchain-text-splitters; the word search needs no extra library, because it is written out in code/hybrid_search.py. Switch on your Python environment, then set four settings: LLM_BASE_URL (the address of your provider’s OpenAI-compatible endpoint), LLM_API_KEY (your key, kept out of any file you share), LLM_MODEL (the chat model name, used by the reranker and the rewrite) and EMBED_MODEL (an embedding model name your provider offers). Then build the index with python code/build_index.py, which prints how many chunks it saved to policy_index/ (our count is in code/README.md, Step 2). Recap of session 17: you compare search settings by a hit count, the number of test questions whose answer words appear in the returned chunks. Have paper or a notes app ready for the Flashcards and the Model answer.
Summary: how do hybrid search, reranking and query rewriting improve a RAG search?
Search by meaning can miss an exact label such as “Learning Policy 5.2”, because the label carries little meaning to match. A word search, written by hand here, finds such labels, and reciprocal rank fusion merges its list with the meaning search into one hybrid list. A reranker, a chat model that reads the question and each candidate together, can move a right chunk that was ranked fifth up to first, and a query rewrite asks the question three ways and fuses the results. Each one costs something, so you keep it only when the hit count rises: in our run (the printed results are in code/README.md, Steps 3 to 5) hybrid scored 14 of 15, hybrid with a rerank scored 15 of 15, and hybrid with rewrites stayed at 14 of 15.
Key concepts: word search, hybrid search, fusion, reranking and query rewriting defined
| Term | Definition |
|---|---|
| Meaning search | Search by embedding: the chunks whose numbers sit closest to the question’s numbers (meaning_search, as in session 14). |
| Word search | A search that scores each chunk by the question’s words it contains, weighting rare words more, so exact labels such as 4.2 can match (keyword_search). |
| Stop words | Very common words such as the, my and what that a word search drops because they do not help it. |
| Hybrid search | A search that combines a keyword signal with a meaning signal so results benefit from both. |
| Reciprocal rank fusion | A way to merge ranked lists: a chunk earns 1 / (60 + rank) from each list it is in, and the earnings are added; the constant 60 is the value set in code/hybrid_search.py, step 6. |
| Reranking | A second stage: search wide for candidates, then have a model read the question and each candidate together and put them in a better order. |
| Query rewriting (multi-query) | Having a chat model write other versions of the question, searching all of them and fusing the lists. |
| Hit | A question whose phrase is inside one of the top three chunks a search returns (TOP = 3). |
Explanations: how do you add and judge each search upgrade?
Why does a search by meaning miss an exact label?
Look at the real table that python code/hybrid_search.py prints (ours is in code/README.md, Step 3, and below). Learning Policy 5.2 is a label, and the meaning search missed it. The opposite also happens: “Who signs off my time off?” shares no word with the clause “Your manager approves or declines a request”, so the word search missed it and the meaning search saved it. Two searches, two kinds of miss.
Now the cause. A catalogue by topic finds the subject quickly, but ask it for “section 5.2” and it has nothing to go on: a clause number has almost no meaning. Meaning search is the same. Pinecone’s guide to hybrid search says each of the two methods misses what the other catches: keyword methods match specific terms such as codes and names, and meaning search matches text phrased with different words.
How does the word search work, and how is it written by hand?
A word index at the back of a book finds the exact term wherever it appears. The word search in code/hybrid_search.py does the same in two small steps. First, words() cuts text into lower-case words and drops the stop words, and a clause number such as 4.2 stays one word, so it can match:
def words(text):
# "4.2" stays one word, so a clause number can match.
return [w for w in re.findall(
r"[a-z0-9]+(?:\.[0-9]+)*", text.lower())
if w not in STOP_WORDS]
A chunk is searched by its policy name, heading and text. Second, a chunk scores the sum, over the question words it holds, of how rare each word is:
for word in set(words(question)):
if word in chunk_words[position]:
how_many_chunks = sum(1 for other in chunk_words
if word in other)
score += math.log(len(chunks) / how_many_chunks) + 1
A word found in few chunks, such as 4.2, counts for more than a word found in many, such as leave. This is a simple rarity weighting, simpler than BM25, a standard word-search score that also weighs how often a word repeats in a chunk and how long the chunk is; this function does neither. No rank_bm25 or other extra library is used; this is the whole word search.
What is hybrid search and how does fusion merge two lists?
Two witnesses to an event: one remembers what was said, the other the exact words. Hybrid search asks both and merges the answers. Reciprocal rank fusion is the merge used here: a chunk earns 1 / (60 + rank) from each list it is in, and the earnings are added, so a chunk near the top of both lists wins. The constant 60 is the value set in fuse in code/hybrid_search.py. Pinecone’s guide notes that fusing separate searches with reciprocal rank fusion is one way to build a hybrid, and that fusion is a method, not a synonym for hybrid search. Because only ranks are used, the two lists do not need to share a score scale.
def fuse(lists, k=TOP):
earned = {}
for ranked in lists:
for rank, chunk in enumerate(ranked, start=1):
key = (chunk.metadata["source"], chunk.page_content)
earned[key] = earned.get(key, 0) + 1 / (60 + rank)
...
def hybrid_search(question, k=TOP):
return fuse([meaning_search(question),
keyword_search(question)], k)
Run python code/hybrid_search.py. Ours printed (rows trimmed):
question meaning keyword hybrid
What does clause 4.2 of the l... hit hit hit
Who signs off my time off? hit MISS hit
Can I take time off to learn ... hit MISS MISS
Learning Policy 5.2 MISS hit hit
...
hits out of 15 14 13 14
A search hits when one of its top three chunks holds the words in the case’s phrase. Read the table by row, not by total: the totals 14, 13 and 14 hide which questions each search saves. Our embedding model is strong, so meaning search alone missed only one label here; yours may miss more or fewer. And “Can I take time off to learn something?” is missed by the fused list too: the right chunk was in the six candidates but ranked below three others. That is what a rerank is for.
What is reranking and how does a chat model do it?
A shortlist, then interviews: a quick look at many, a closer look at a few. Reranking is a second stage. Pinecone describes it as querying an index for a number of results and then sending the query and the results to a reranking model that returns a more accurate ranking. In code/rerank.py the reranker is a chat model, not a special service, and no Cohere or other rerank service is used. The hybrid search hands over six candidates (CANDIDATES = 6), the model gives each a score from 0 to 10 for how directly it answers the question, and the best three are kept (KEEP = 3):
class Scores(BaseModel):
scores: list[int] = Field(description=...)
scorer = prompt | model.with_structured_output(Scores)
scores = scorer.invoke({"question": question,
"pieces": pieces}).scores
order = sorted(range(len(candidates)),
key=lambda i: (-scores[i], i))
return [candidates[i] for i in order[:keep]]
The reply is a list of numbers, one per chunk, in order; if the model returns too few, the missing scores count as 0, and ties keep the first search’s order. Run python code/rerank.py. Ours printed (trimmed):
Question: Can I take time off to learn something?
hybrid order:
1. learning-policy.md / 1. Purpose
...
5. learning-policy.md / 4. Learning days
after rerank:
1. learning-policy.md / 4. Learning days
2. learning-policy.md / 1. Purpose
Hybrid top 3: 14 of 15 hits
Hybrid + rerank, top 3: 15 of 15 hits
The chunk about learning days was fifth; the model, reading the question and the chunk together, moved it to first. The cost is one more chat call per question.
What is query rewriting and when does it help?
Asking a librarian in their own terms: “annual leave carry-over” may find more than “my leftover days”. code/rewrite_query.py has a chat model write three other versions of the question (code/model_prompts/rewrite.md tells it to keep any clause number or policy name), searches the original and each version with the hybrid search, and fuses the lists. LangChain’s MultiQueryRetriever has the same idea, as its documentation puts it: use an LLM to write a set of queries, retrieve for each and return the unique union; here it is spelled out. Run python code/rewrite_query.py. Ours printed (trimmed):
Question: What happens to my days when the year ends?
rewrite 1: What is the policy regarding the carryover
of unused leave days at the end of ...
rewrite 2: How are accrued vacation days handled ...
results:
- leave-policy.md / 4. Annual leave
Hybrid top 3: 14 of 15 hits
Hybrid + rewrites, top 3: 14 of 15 hits
The rewrites are good, but the hit count did not move: the one missed question is still missed. A rewrite helps when your words differ from the document’s; it does not help when the right chunk is already found but ranked low, which is the rerank’s job. It costs a chat call per question and one more search per rewrite, so measure before you keep it. A related idea from the research literature, writing a made-up answer and searching with it, is theory only: this code does not do it.
Three situations where search needs help
Learning Policy 5.2is missed by meaning search. The label carries little meaning to match, and only the word search found it.- The right chunk is in the six candidates but ranked fifth. The first search ordered results roughly, and the rerank, reading question and chunk together, moved it to first.
- The rewrites are good, yet the hit count does not move. The missed question was found but ranked low, which a rewrite does not fix.
| What you learned | The design decision | Where it shows |
|---|---|---|
| A label has little meaning to match | Add a word search beside the meaning search | keyword_search |
| Two lists must become one | Fuse by rank, so no shared score scale is needed | fuse |
| A right chunk can be ranked low | Rerank six candidates, keep three | code/rerank.py |
| An upgrade can cost more than it gives | Keep it only if the hit count rises | 14 against 15 of 15 |
Basic assignment: in code/rerank.py set KEEP = 1 and say whether the hit count stays at 15. In code/hybrid_search.py set TOP = 1 and see how the three searches compare. Remove the + 1 in the word score, or change the constant 60 in fuse: does a row change?
Project step: keep what raises the count. Run python code/build_index.py, python code/hybrid_search.py, python code/rerank.py and python code/rewrite_query.py in order and write down each hit count. Say which upgrade you would keep and which you would drop. The build prompts are code/prompts/1-write-hybrid-search.md, code/prompts/2-write-rerank.md and code/prompts/3-write-rewrite-query.md. Stop when all three scripts print their counts.
Practice: explain and diagnose. First, answer the Model answer question below aloud in four beats before reading it. Second, for each situation, say which upgrade you would try:
- The question names “Expenses Policy 3.5” and the meaning search returns other expense clauses.
- The right chunk is among the candidates but ranked fifth.
- The employee’s words share nothing with the policy’s words.
Flashcards: can you answer these hybrid search questions aloud?
Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.
1. Why can a meaning search miss "Learning Policy 5.2"?
A clause label carries almost no meaning, so there is little for an embedding to match. A word search finds the exact words instead.2. How does the hand-written word search score a chunk?
It adds up, over the question words the chunk contains, how rare each word is across all chunks. A rare word such as `4.2` counts for more than a common one such as `leave`.3. Why does `words()` keep `4.2` as one word?
So a clause number can match as a whole, as the comment in the code says, and a question that names clause 4.2 can find the chunk that holds that exact number.4. What is hybrid search?
A search that combines a keyword signal with a meaning signal, so results benefit from both. Here the two ranked lists are merged by reciprocal rank fusion.5. How does reciprocal rank fusion score a chunk?
A chunk earns `1 / (60 + rank)` from each list it is in, and the earnings are added. A chunk near the top of both lists wins, and no shared score scale is needed.6. Why read the hit table by row and not by total?
The totals can hide which questions each search saves. `Learning Policy 5.2` was saved by the word search and "Who signs off my time off?" by the meaning search.7. What does reranking do, and what does it cost?
It takes six hybrid candidates, has a chat model score each from 0 to 10 against the question, and keeps the best three. It costs one more chat call per question.8. What did the reranker change in our run?
The chunk about learning days moved from fifth to first, and the hit count rose from 14 of 15 to 15 of 15.9. What is query rewriting and did it help in our run?
A chat model writes three other versions of the question, and the lists from all four are fused. In our run the count stayed at 14 of 15, so it was not worth its extra calls there.10. How do you decide to keep an upgrade?
Count hits on the same fifteen questions with and without it, and keep it only when the count rises enough to justify its extra calls or code.Model answer: how do you explain improving a RAG search beyond meaning search?
The core question this session answers, as an examiner might put it: “Meaning search sometimes misses. What else can you add to a RAG search, and how do you decide to keep it?”
Four beats, in this order: word search and fusion · rerank · rewrite · decide by hits. Skip any of these four and the answer is incomplete.
- Word search and fusion: a hand-written word search finds exact labels, and reciprocal rank fusion merges it with the meaning search.
- Rerank: a chat model scores six candidates against the question and keeps the best three, at the cost of one more call.
- Rewrite: a chat model asks the question three ways, and the lists are fused.
- Decide by hits: keep an upgrade only when the hit count on the 15 questions rises.
The full answer, as you would say it to the examiner:
(Word search and fusion) “Meaning search can miss an exact label like a clause number, so I add a word search that scores rare words higher and merge the two ranked lists by reciprocal rank fusion.
(Rerank) If the right chunk is found but ranked low, I search wide for six candidates and have a chat model score each against the question, keeping the best three. That costs one more call.
(Rewrite) If the employee’s words differ from the policy’s, a chat model writes the question three ways and I fuse the lists.
(Decide by hits) I count hits on the same fifteen questions with and without each upgrade and keep one only when the count rises. In our run the rerank did, and the rewrite did not.“
Missing a beat is a failure state. The follow-up question lands where you went quiet: skip the last beat and the next question is how you know it was worth it. Practise until the four come out without prompting.
Why this matters: where does hybrid search matter in real systems?
In a support team: people quote section numbers and product codes, and a pure meaning search can miss them. A word search beside it is a small, readable fix.
On the job: each upgrade adds a call or a script to maintain. A hit count on a fixed set of questions lets you say what you got for that cost, which is the review question that matters.
In the rest of this course: session 19 is the Milestone 1 lab, where you assemble a copilot that answers from documents with sources, and the evaluation sessions grow the test set into a proper measure.
After this unit: what to build after you add hybrid search
Assignment. In code/rerank.py set KEEP = 1 and say whether the hit count stays at 15. The brief for the project work, keep what raises the count, is the project file. Keep your index: later sessions search it.
What’s next. Session 19 is the lab and Milestone 1: a copilot that answers from documents, with sources, checked by a test set.
Want help building this with someone?
Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.
Ask about mentor-led trainingTechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.