Chapter 1 — Embeddings and Vector Search

Course: Build and Evaluate RAG Applications

Prerequisites: The course prerequisite, Getting Started with AI. You can read and run a short Python script, and you can work with lists of numbers in code. From that course it helps to know that a model reads its input from a limited context window, but the notes define that again before they use it. No linear algebra beyond multiplying and adding numbers is assumed: vectors, cosine similarity, nearest-neighbour search and recall are all defined in the notes before they are used.

What the reader can do after this chapter:

• Compute the cosine similarity between a query vector and a set of document vectors, rank the documents by it, and say what a high score does and does not tell you about a document.
• Diagnose a vector-search miss from its symptoms: an exact identifier that a dense ranking buries, a negated query that returns the opposite fact, or a result list cut short by approximate search.
• Combine a lexical ranking and a dense ranking with reciprocal rank fusion, and measure whether the change helped with recall at a fixed cut-off.

Core question this chapter answers: When a vector search returns a passage as “most similar” to a query, what has actually been measured, and in which situations does that measurement fail to find the passage that answers the question?

Connections:

• Builds on: a lesson in Getting Started with AI that put model calls behind one interface. Embedding calls belong behind the same interface, and this chapter shows why the embedding model’s version has to be pinned with the index.
• Leads into: Chapter 2, which builds a full retrieval-augmented generation pipeline. It opens with the small set of test questions, each paired with the passage that answers it, that you write in this chapter’s project, and uses them to measure every stage of the pipeline.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question, so you can check that your answer has the right shape before you submit it.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines. Every line’s accepted value, or the rule that decides it, follows from what the task prints, so you can check your own file before you submit it.

Build and Evaluate RAG Applications — Chapter 1: Embeddings and Vector Search

Before you start

Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library. You do not need an account with any model provider or an API key: every vector the project uses is printed in the project file.

Keep one question in mind while you read: when a search tool tells me a passage is the “closest match”, closest in what sense? By the end of the chapter you should be able to answer it with a number, and to name the situations where that number points at the wrong passage.

Summary

An embedding turns a piece of text into a fixed-length list of numbers, a vector, and the distance between two vectors is used as a measure of how related the two texts are. Vector search embeds the documents once, embeds each query when it arrives, and returns the documents whose vectors are closest to the query’s, often measured by cosine similarity. That measurement is relatedness, not correctness: it can rank a passage that says the opposite of what was asked above the one that answers it, it can bury a passage that contains the exact error code the user typed, and when the index uses approximate search it can leave the closest passage out of the results altogether. The engineering response is to measure retrieval with a small set of questions whose answers you know, and to combine dense search with lexical search, filters and reranking where the measurement shows dense search alone falls short. This chapter builds the vector-search half of the retrieval pipeline that the next chapters assemble, and the habit of measuring it.

Key concepts

Term	Definition
Embedding	A vector of floating-point numbers that an embedding model produces for a piece of text, such that the distance between two vectors measures how related the two texts are.
Vector	An ordered list of numbers of fixed length. The length is the vector’s number of dimensions.
Dimensions	How many numbers an embedding has. It is fixed by the embedding model and its settings, and two vectors can only be compared if they have the same number of dimensions.
Cosine similarity	A score for how closely two vectors point the same way: their dot product divided by the product of their lengths. It runs from -1 to 1, and higher means more related.
Dot product	The sum of the products of two vectors’ matching entries. For vectors already scaled to length 1, it equals their cosine similarity.
Nearest-neighbour search	Finding the k stored vectors with the highest similarity (or smallest distance) to a query vector.
Exact search	Nearest-neighbour search that compares the query with every stored vector, so it always returns the true top k.
Approximate search	Nearest-neighbour search that uses an index structure or compressed vectors to compare against fewer of them, trading some chance of missing a true neighbour for speed or memory.
Recall at k	Of the passages that should have been found, the share that appear in the top k results. Measured against a set of questions whose answers you already know.
Lexical search	Ranking documents by the words they share with the query. BM25 is one lexical ranking function.
Hybrid search	Running lexical and dense (embedding) search side by side and merging their rankings into one list.
Reciprocal rank fusion (RRF)	A rule for merging rankings: each document scores 1 ÷ (k + its rank) in every list it appears in, the scores are added, and the merged list is sorted by the total.

Explanations

The failure, before the explanation

Here is an internal help search in front of a software product’s documentation. It embeds every help article, embeds each question, and returns the two closest articles. Two questions arrive in the same afternoon. (The project file calls the first one q2 and the second one q1.)

The error query: error E-4012

Top two results: “Common error messages and what they mean”, “Troubleshooting failed uploads”

Not in the top two: “Error E-4012 means the upload exceeded the size limit” — the one article that answers it, ranked third.

The plans query: plans without single sign-on

Top result: “Single sign-on is included in the Business and Enterprise plans”

Second: “The Starter plan does not include single sign-on” — the article that answers it.

Nothing crashed, and both result lists look sensible at a glance: every article is on the right topic. But the first search buried the only article that names the user’s exact error, and the second put the opposite fact first. An engineer from a backend or QA background expects a search with a bug to return something unrelated. These results are related, and still wrong. To see why, you need to know what the search measured.

Embeddings: text as a point in space

Start with a concrete case. An embedding model reads a piece of text and returns a list of numbers. One provider’s models return 1,536 numbers for its small embedding model and 3,072 for its large one by default (OpenAI, Vector embeddings). The same documentation gives the definition this chapter works from: “An embedding is a vector (list) of floating point numbers. The distance between two vectors measures their relatedness.”

Those 1,536 numbers are hard to picture, so this chapter uses a teaching example with four. Each of the four positions has been given a label so you can read it — account, plans, errors, billing — and each article has been given a value for each. These are hand-made teaching vectors, not the output of any real model; the documentation describes a real embedding only as a list of numbers, with no labels on its positions.

Article	account	plans	errors	billing
d3 “Error E-4012 means the upload exceeded the size limit”	0.1	0.0	0.6	0.3
d4 “Common error messages and what they mean”	0.2	0.1	0.9	0.1
query error E-4012	0.25	0.05	0.85	0.05

A second way to picture it: a colour picker. Any colour on a screen can be written as three numbers for red, green and blue, and two shades of orange have numbers that are close together even though their names (“tangerine”, “amber”) share no letters. An embedding does the same for text, with many more numbers, so that two sentences about the same thing get nearby vectors even when they share no words. That is the strength of embeddings. This chapter is mostly about the cost of it.

One consequence for your system design follows straight from the definition. Vectors can only be compared if they come from the same space. Two models with different default lengths, 1,536 and 3,072, cannot even be compared number by number, and nothing in the documentation says vectors from two different models can be mixed in one index. Treat the embedding model and its version as part of the index: pin them behind the same interface you use for every other model call, and plan to re-embed the stored documents if you change the model.

Measuring closeness: cosine similarity, by hand

Now the measurement itself, with the numbers from the table.

Cosine similarity multiplies the two vectors entry by entry, adds the products (the dot product), and divides by the product of the two vectors’ lengths, where a vector’s length is the square root of the sum of its squared entries. For the query and d4:

• Dot product: 0.25×0.2 + 0.05×0.1 + 0.85×0.9 + 0.05×0.1 = 0.825
• Query length: √(0.25² + 0.05² + 0.85² + 0.05²) ≈ 0.8888
• d4 length: √(0.2² + 0.1² + 0.9² + 0.1²) ≈ 0.9327
• Cosine: 0.825 ÷ (0.8888 × 0.9327) ≈ 0.995

The same sum for d3 gives about 0.912. So by this measure d4, the general article about error messages, is closer to error E-4012 than d3, the article about that exact error. That is the first failure from the help search, reproduced with arithmetic.

A smaller example to check your understanding: the vectors [3, 4] and [4, 3] have a dot product of 24, and each has length 5, so their cosine similarity is 24 ÷ 25 = 0.96. The vectors [3, 4] and [6, 8] point the same way and score exactly 1, even though one is twice as long: cosine looks at direction and ignores length.

A second way to picture it: two arrows drawn from the same point. Cosine similarity asks only how small the angle between them is. Two arrows pointing in the same direction score 1, arrows at right angles score 0, and arrows pointing in opposite directions score -1.

Providers recommend a specific measure for their own models. One provider says “We recommend cosine similarity”, notes that its embeddings “are normalized to length 1”, and draws two consequences: cosine similarity can be computed “slightly faster using just a dot product”, and “Cosine similarity and Euclidean distance will result in the identical rankings” (OpenAI, Vector embeddings). Check whether the model you call scales its vectors to length 1 before you replace the cosine formula with a bare dot product.

What “similar” measures, and what it does not

The documentation’s own word is relatedness. It is worth taking that word literally, because it excludes three things engineers often assume a high score means.

It does not mean “answers the question”. In one open-source library’s terms, the search finds the documents whose embeddings are closest to the query’s: “the query is embedded into the same vector space and the closest embeddings from your corpus are found” (Sentence Transformers, Semantic Search). A general article about error messages is closely related to a question about one error, so it scores high. Whether it contains the answer is a different question that the score does not ask. The same library separates symmetric search, where query and document are similar in length and content (two phrasings of the same question), from asymmetric search, where a short query has to find a longer passage, and says that choosing the right model for the case is critical. A help search is the asymmetric case.

It does not reliably capture negation. A research study built a benchmark of document pairs that differ only by a negation and found that most retrieval models “do not consider negation, performing the same or worse than a random ranking”. Among the architectures tested, cross-encoders performed best, and bi-encoders — the kind that embed query and document separately, as vector search does — were in last place, alongside sparse neural models (Weller, Lawrie and Van Durme, NevIR: Negation in Neural Information Retrieval). That is the second help-search failure: plans without single sign-on sits close to both the article that includes SSO and the one that does not. In the teaching vectors the two articles score 0.997 and 0.996 — the word not barely moves the point.

It does not guarantee an exact match on an identifier. One provider’s write-up of its retrieval experiments gives this case: “An embedding model might find content about error codes in general, but could miss the exact ‘TS-999’ match. BM25 looks for this specific text string.” (Anthropic, Introducing Contextual Retrieval). An error code, a product SKU, a ticket number or a function name is the kind of string that a lexical search matches directly and that a dense search can rank below a general article.

Finding the nearest: exact and approximate search

Picture a library with a card catalogue and a user who asks for books about one topic. One librarian reads every card in the catalogue and hands over the best five. A second librarian walks to the shelf the catalogue points at and checks only the books near it. The second is far faster, and now and then misses a book shelved one aisle over. Nothing tells the user a book was missed: they still get five books, just not always the best five.

Those two librarians are the two ways to search stored vectors. The first is exact search: compare the query with every stored vector and keep the top k. It is what the project does with seven articles, and it always returns the true top k. The second is approximate search: use an index structure or compressed vectors to compare against fewer of them, trading some chance of missing a true neighbour for speed or memory.

With millions of vectors, comparing against every one is expensive, so similarity-search libraries offer approximate methods. One such library offers both exact and “approximate (compressed-domain) nearest neighbor search”, and describes its index types as “various trade-offs with respect to search time, search quality, memory used per index vector” and more (Faiss, project README). One such index type, a graph index called HNSW, exposes a search setting called ef, and its documentation states the trade plainly: “Higher ef leads to more accurate but slower search” (hnswlib, HNSW algorithm parameters).

A second picture, from operations: sampling metrics instead of recording every request. You get the answer faster and cheaper, it is often close enough, and you only find out what you lost if you compare against a full measurement. For a vector index, that comparison is recall at k.

Measuring retrieval: recall at k

Take the help search from the start of the chapter and ask: did it find the article that answers each question? For the error query, the answering article d3 sat third, so a search that returns only two articles missed it. For the plans query, the answering article d5 sat second, so a search that returns two articles found it. That is recall at 2, one question at a time: 0 for the first, 1 for the second.

A second example from outside search: a spam filter’s test set. You do not judge a filter by reading a few messages it caught; you hold a set of messages whose labels you know and count how many it got right. A set of questions with known answers is the same thing for retrieval, and it is the artifact this chapter’s project builds. You cannot tell whether a search works by looking at its results one query at a time, because a result can look related and still be the wrong one.

Now the definition. Recall at k is, of the passages that should have been found, the share that appear in the top k results. With one correct passage per question, it is the share of questions whose correct passage appears in the top k. One provider’s retrieval write-up reports its experiments in exactly this form: it uses “1 minus recall@20” as its metric, which “measures the percentage of relevant documents that fail to be retrieved” (Anthropic, Introducing Contextual Retrieval).

The plans query is also a warning about the metric. Its recall at 2 is 1, yet the article ranked first is the wrong one. Recall at 2 says the answer was found; it says nothing about the wrong article sitting above it. Recall tells you whether the answer reached the shortlist. Whether the model then reads the right item from the shortlist is a later chapter’s problem.

Hybrid search: two rankings, one list

Start with the error query again. The dense search ranked the general article d4 first, the upload troubleshooting article d7 second and the exact article d3 third. A word-matching search over the same seven titles would find d3 first, because it is the only title containing e-4012, and d4 second, because it contains error. Each ranking has half of what you want. Running both and merging them is hybrid search: lexical and dense (embedding) search side by side, their rankings merged into one list.

One provider describes the steps it used: “Use BM25 to find top chunks based on exact matches; Use embeddings to find top chunks based on semantic similarity; Combine and deduplicate results … using rank fusion techniques” (Anthropic, Introducing Contextual Retrieval). In its experiments, combining its contextual embeddings with a contextual BM25 index “reduced the top-20-chunk retrieval failure rate by 49% (5.7% → 2.9%)”, and combining those with reranking cut it by 67% (5.7% → 1.9%). Those numbers are for that provider’s datasets and settings; they show the direction of the effect, and your own measurement gives your numbers.

Reciprocal rank fusion is one common way to merge the lists. Each document scores 1 ÷ (k + rank) in each list it appears in, and the scores are added; one search engine’s documentation defines it this way and sets the constant k to 60 by default (Elastic, Reciprocal rank fusion). Worked on the error query:

Article	Dense rank	Lexical rank	RRF score
d4 “Common error messages…”	1	2 (shares error)	1/61 + 1/62 ≈ 0.03252
d3 “Error E-4012 means…”	3	1 (shares error, e-4012)	1/63 + 1/61 ≈ 0.03227
d7 “Troubleshooting failed uploads”	2	not returned	1/62 ≈ 0.01613

A second way to see it: two judges each rank the same entrants, and an entrant earns points for every judge who placed them, more for a high place. Someone both judges placed reasonably well can beat someone only one judge loved. Here d3 is placed first by the lexical judge and third by the dense judge, so it ends close behind d4.

The merged top two are d4 and d3. Recall at 2 for this query goes from 0 to 1. The general article is still first, but the answer is now in the shortlist, which is the job of this stage.

Now the honest result on the plans query: the lexical search matches single, sign-on and plans in the SSO-included article and only single and sign-on in the Starter article, so it agrees with the dense search and the wrong article stays first. In this example, hybrid search leaves the negation failure in place. When the question is about an attribute such as which plan includes a feature, one fix that does not depend on ranking is to store the attribute as data. One vector database’s documentation describes attaching fields to stored records, so that you can “include a metadata filter to limit the search to records matching the filter expression” (Pinecone, Filter by metadata). A filter on includes_sso = false removes the article that includes SSO from the plans query’s results, because the condition is checked as data rather than inferred from wording.

Three situations you will meet, and what they mean for your design

Before the table, three situations:

1. A support engineer pastes a stack trace’s exception name into the internal search, and the top results are general troubleshooting pages. The page about that exact exception is on page two.
2. A sales engineer asks the assistant which regions a service is not available in, and gets a confident list of the regions where it is available, drawn from the right document.
3. A platform team moves its index to a faster approximate setting before a launch. Nobody reports an error, but answers start citing slightly less relevant passages.

What you see	What the search measured	What to do instead
An exact identifier buried below general pages (situation 1)	Relatedness of meaning; the identifier is a small part of the text	Add lexical search and fuse the rankings; measure recall on identifier queries
The opposite fact returned for a negated question (situation 2)	Closeness that barely moves for not	Store the attribute as a field and filter on it; do not rely on ranking
Quietly worse results after an index change (situation 3)	Fewer comparisons, so some true neighbours skipped	Re-run recall at k on the known questions before and after any index or model change

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

What does the distance between two embeddings measure?

How related the two texts are. One provider’s documentation says exactly that: the distance between two vectors measures their relatedness. It does not measure whether a passage answers a question or whether it is true.

How do you compute cosine similarity?

Multiply the two vectors entry by entry and add the products (the dot product), then divide by the product of the two vectors’ lengths. For [3, 4] and [4, 3]: 24 ÷ (5 × 5) = 0.96.

Why can [3, 4] and [6, 8] have a cosine similarity of exactly 1?

They point in the same direction. Cosine similarity measures the angle between two vectors and ignores their lengths, so doubling a vector does not change its score.

When is a dot product the same as cosine similarity?

When both vectors are already scaled to length 1. One provider says its embeddings are normalised that way, so a dot product gives the cosine and ranks results the same as Euclidean distance. Check your model before you rely on it.

Why can a search for an exact error code return general error pages first?

Dense search ranks by overall relatedness of meaning, and a general page about errors is closely related to any error query. The exact code is a small part of the text. Lexical search, such as BM25, looks for the exact string.

Does hybrid search fix a negated question like "plans without SSO"?

Not reliably. Both the dense and the lexical ranking match the words of the topic, not the negation. Store the attribute as a field on each record and filter on it.

What does approximate nearest-neighbour search trade away?

Some chance of returning the true nearest neighbours, in exchange for speed or memory. It still returns k results, so the loss is silent; only a recall measurement shows it. In one graph index, a higher search setting means more accurate but slower search.

What is recall at k, with one correct passage per question?

The share of test questions whose correct passage appears in the top k results. A question whose answer is ranked third has recall at 2 of 0 and recall at 3 of 1.

How does reciprocal rank fusion score a document?

It adds 1 ÷ (k + rank) for every list the document appears in, with k set to 60 by default in one search engine. A document ranked first in one list and third in another scores 1/61 + 1/63.

What must you do to your stored vectors if you change embedding models?

Re-embed them with the new model. Vectors are only comparable within one model’s space; two models can even produce different numbers of dimensions. Treat the model and its version as part of the index.

Model answer

The question: “Our internal search uses embeddings, and users say it sometimes misses the obvious answer. How does vector search decide what is ‘most similar’, and how would you find out where it is failing?” This is the question a lead engineer asks before approving a retrieval design, and an interviewer for a role building with models asks to see whether you can debug one. A strong answer hits five beats, in this order:

1. Embedding — each text, query and document alike, is turned into a vector by the same embedding model.
2. Similarity — documents are ranked by a closeness score, commonly cosine similarity, which measures relatedness, not whether the document answers the question.
3. Blind spots — that score is weak on exact identifiers and on negation, and approximate indexes can silently skip true neighbours.
4. Measurement — you find the failures with a set of questions whose correct passages you know, and compute recall at k.
5. Remedy — you choose a fix that matches the failure: lexical search fused with the dense ranking for identifiers, a metadata filter for attributes, a higher-accuracy index setting for approximate misses, and you re-measure.

“Every document was turned into a vector by an embedding model when we indexed it, and every query is turned into a vector by the same model when it arrives. The search then ranks documents by how closely their vectors point the same way as the query’s — cosine similarity. That score measures relatedness, which is why a general troubleshooting page can outrank the one page that names the user’s exact error code, and why a question with not in it can return the opposite fact: both are closely related. If the index uses approximate search, it can also skip a true neighbour without telling us. So I wouldn’t debug it by eyeballing results, because a related-looking result can still be wrong. I’d build a small set of real questions with the passage that answers each, and measure recall at the cut-off we actually pass to the model. Then I’d match the fix to the failure: fuse a lexical ranking with the dense one for identifiers, filter on a stored field for attributes like plan or region, and raise the index’s accuracy setting if approximate search is the cause — and re-run the same questions after each change.”

Missing a beat is a failure state. Without embedding the answer cannot explain why the model version belongs to the index. Without similarity it cannot say what the score means. Without blind spots “it misses things” stays a mystery. Without measurement every fix is a guess. Without remedy the diagnosis does not change anything.

Why this matters

• Retrieval sits under the grounded answers later chapters build. Later chapters put retrieved passages into a model’s context window and ask it to answer from them. A passage the search never returned cannot be cited, however good the model is.
• Customers search with identifiers. Error codes, part numbers, ticket ids and function names appear in real queries, and a dense search on its own can rank them poorly, as the first failure in this chapter shows.
• Index changes are deployments. Changing the embedding model means re-embedding the stored documents, and changing an approximate index’s settings changes what is found. Both need the same before-and-after measurement as any other release.
• Talking to customers. When a customer says “your search ignored the word not”, the engineer who can explain relatedness, show the measurement and propose a filter keeps the conversation technical rather than defensive.

After this chapter

The project for this chapter is Measure a Search Before You Trust It (project.md). You will rank the help articles from these notes by cosine similarity, add a lexical ranking and merge the two with reciprocal rank fusion, measure recall at 2 before and after, and finish by writing a small set of test questions, each paired with the passage that answers it, from documentation you actually work with.

Chapter 2 starts from that set. It splits your passages into chunks, indexes them, retrieves for each question and assembles the result into a model’s context, and uses your questions to measure each of those stages.

Hands-on Project — Measure a Search Before You Trust It

Objective

Rank a small set of help articles by cosine similarity, add a lexical ranking and merge the two with reciprocal rank fusion, measure recall at 2 before and after, and write a set of test questions from your own documentation, so that you can judge a vector search by measurement rather than by how its results look.

What this project needs: Python 3 and its standard library only (math and re). No task needs an account with a model provider, an API key, a vector database or any payment. The vectors in Tasks 1 and 2 are hand-made teaching values, printed below, so every result can be checked exactly. All three tasks are required.

Your evidence file. Keep one plain-text file, chapter1-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print. Where a value is a list of article ids, write them comma-separated with no spaces, in the order your code produced them.

The data for Tasks 1 and 2

Copy this into your Python file. It is the help index from the notes: seven articles, two queries, and for each query the id of the article that answers it (its gold article).

ARTICLES = {
    "d1": ("Reset your password from the sign-in page", [0.9, 0.1, 0.2, 0.0]),
    "d2": ("Single sign-on is included in the Business and Enterprise plans", [0.6, 0.8, 0.0, 0.1]),
    "d3": ("Error E-4012 means the upload exceeded the size limit", [0.1, 0.0, 0.6, 0.3]),
    "d4": ("Common error messages and what they mean", [0.2, 0.1, 0.9, 0.1]),
    "d5": ("The Starter plan does not include single sign-on", [0.5, 0.7, 0.1, 0.2]),
    "d6": ("Invoices are emailed on the first day of each billing cycle", [0.0, 0.3, 0.0, 0.9]),
    "d7": ("Troubleshooting failed uploads", [0.1, 0.0, 0.8, 0.1]),
}

QUERIES = {
    "q1": ("plans without single sign-on", [0.55, 0.75, 0.05, 0.15], "d5"),
    "q2": ("error E-4012", [0.25, 0.05, 0.85, 0.05], "d3"),
}

Task 1 — Rank by cosine similarity

Write these two functions.

1. cosine(a, b) — the dot product of a and b divided by the product of their lengths, where a vector’s length is the square root of the sum of its squared entries. Before going further, print cosine([3, 4], [4, 3]) to 3 decimal places and check it against the notes.
2. dense_rank(query_vector) — returns all seven article ids sorted by cosine similarity with the query vector, highest first. If two scores are exactly equal, put the smaller id first.

Then, for query q2:

3. Print its dense ranking.
4. Print the cosine score of the top-ranked article, and the cosine score of its gold article d3, each to 3 decimal places.

Stop there: one query, seven articles. You do not need numpy, a vector database or a real model.

Add to your evidence file:

t1_cos_check: <cosine([3, 4], [4, 3]), 3 decimal places>
t1_q2_dense: <the seven ids, highest first>
t1_q2_top_score: <score of the top-ranked article, 3 decimal places>
t1_q2_gold_score: <score of d3, 3 decimal places>

Output: one Python file with two functions, and four evidence lines. You can check the first and third values yourself: the notes work both out by hand.

Task 2 — Add a lexical ranking and fuse the two

This task stands on its own: it needs only the data above plus the cosine and dense_rank functions, which you can copy from Task 1 or write again.

First, add this word splitter to your file exactly as printed. It lower-cases the text and keeps runs of letters and digits, including hyphenated runs such as sign-on and e-4012:

import re

def words(text):
    return set(re.findall(r"[a-z0-9]+(?:-[a-z0-9]+)*", text.lower()))

Then write:

1. lexical_rank(query_text) — returns the ids of the articles whose title shares at least one word with the query, sorted by the number of shared words, most first. If two articles share the same number, put the smaller id first. Articles that share no words are not in the list. (This counts shared words, which is a simpler rule than BM25, the lexical function named in the notes; the simpler count is enough to see the effect here.)
2. fuse(rankings, k=60) — reciprocal rank fusion. For each ranking, an article at position r (counting from 1) gains 1 ÷ (k + r). An article missing from a ranking gains nothing from it. Add the gains, and return the ids sorted by total, highest first, smaller id first on an exact tie.
3. recall_at(k, ranking, gold) — returns 1 if gold is among the first k ids of ranking, otherwise 0.

For each of the two queries, compute the dense ranking, the lexical ranking and the fused ranking of the two. Then compute recall at 2 for the dense ranking and for the fused ranking.

Add to your evidence file:

t2_q2_lexical: <lexical ranking for q2>
t2_q2_hybrid_top2: <first two ids of the fused ranking for q2>
t2_q2_recall2_dense: <0 or 1>
t2_q2_recall2_hybrid: <0 or 1>
t2_q1_lexical: <lexical ranking for q1>
t2_q1_hybrid_top2: <first two ids of the fused ranking for q1>
t2_q1_recall2_hybrid: <0 or 1>
t2_mean_recall2_dense: <mean of the two queries' dense recall at 2, 2 decimal places>
t2_mean_recall2_hybrid: <mean of the two queries' fused recall at 2, 2 decimal places>

Before you submit, look at where the answering article sits in each of your rankings for q1. Write one sentence for yourself on what recall at 2 does not tell you about that position; it is not scored.

Output: a lexical ranking, a fusion function and a recall function added to your file, and nine evidence lines.

Task 3 — Write test questions from documentation

Choose your source. Option A (your own documentation): a runbook, an internal wiki section, a product’s help pages or a project’s README files; collect eight to twenty short passages from it, a paragraph each. Option B (no documentation to hand): use the eight passages below, which are invented for this exercise and describe a fictional command-line tool called syncer. Save each passage as its own text file in a folder called passages/, named p01.txt, p02.txt and so on.

p01: The syncer tool copies files from a local folder to a remote store. Run `syncer push ./reports` to upload every changed file in the folder.
p02: The syncer tool reads its settings from a file named syncer.toml in the project root. The key remote_url sets the address of the remote store.
p03: A failed upload is reported with exit code E-207 and the message "quota exceeded". Free up space in the remote store, then run the push again.
p04: The syncer tool does not follow symbolic links. A link inside a pushed folder is skipped, and a warning is printed for each one.
p05: Run `syncer pull` to download files from the remote store. Files that already exist locally and have not changed are left alone.
p06: The ignore file is named .syncignore. Each line is a pattern; any file matching a pattern is left out of every push.
p07: The syncer tool cannot upload a single file larger than 2 GB. Larger files must be split before they are pushed.
p08: Use the --dry-run flag with any command to list what would change without copying anything.

Then write exactly five test questions, each paired with the one passage that contains its answer. Save them in gold.tsv, one per line, as the question, a tab character, and the passage file name without .txt (this line shows the shape only):

How do I rotate the staging database password?	p12

Each question must contain at least three different whole words of four or more letters that also appear in the passage it is paired with, so that the pairing can be checked by a script. Include at least these two kinds among your five, and mark each by line number in the evidence file:

• one identifier question — it contains a string of at least four characters, with no spaces, that includes a digit, hyphen, underscore, dot or slash (such as an error code, a command flag, a ticket id or a configuration key), and that exact string appears in the passage file the question points to;
• one negation question — it asks what something does not do, not include or not allow, and contains one of the words not, without or cannot.

Stop at five questions. You do not need to embed anything or run a search in this task.

You submit the passages/ folder and gold.tsv along with the evidence file, so the marker can open them. Add to your evidence file:

t3_passages: <number of files in passages/>
t3_gold_rows: <number of lines in gold.tsv>
t3_identifier_row: <the line number, 1 to 5, of your identifier question>
t3_negation_row: <the line number, 1 to 5, of your negation question>

Output: a passages/ folder, a gold.tsv of five question-and-passage pairs, and four evidence lines.

What carries forward

Keep passages/, gold.tsv and chapter1-evidence.txt. The next chapter opens with them: it splits your passages into chunks, indexes them, retrieves for each of your five questions, and measures recall against gold.tsv at every stage of the pipeline. Your identifier and negation questions are the two it watches most closely.