Courses / Agentic AI from Scratch (Python)
Session 14 of 40
Vector databases: Chroma, and FAISS, pgvector and Pinecone compared
Notes
What Is a Vector Database and When Should You Use Chroma, FAISS, pgvector or Pinecone?
Agentic AI Foundations · Session 14 — Vector databases: Chroma, and FAISS, pgvector and Pinecone compared
Before you start: what do you need to store and search vectors with Chroma?
You need the code folder for this session, chapters/14-chroma-index/, open in your terminal. It holds build_index.py, search_index.py, load_policies.py (session 12’s loader and splitter, carried over), data/policies/ with the 5 policies, and two prompt files in prompts/. Install the libraries with pip install langchain-chroma langchain-openai langchain-community langchain-text-splitters. The settings are the same three as session 13: LLM_BASE_URL, LLM_API_KEY and EMBED_MODEL; set them in both scripts, or once in your terminal. The folder policy_index/ is not in the lesson files: you make it in this session, and it is yours to delete and rebuild. Have paper or a notes app ready for the Flashcards and the Model answer.
Summary: what is a vector database and how do Chroma, FAISS, pgvector and Pinecone differ?
A vector database stores embeddings together with their text and metadata and finds the stored items closest to a question’s numbers, which is how the helper searches the policies by meaning. Session 13 embedded all 34 chunks on every run. This session builds a Chroma index on disk once, with build_index.py, and search_index.py opens the saved folder and embeds only the question. Chroma prints a distance, where a smaller number means a closer chunk. The saved index is a copy of the policy files, so after a policy changes the index is rebuilt. FAISS, pgvector and Pinecone are compared as theory only: they differ mostly in where they run and who looks after them, and the code in this session uses Chroma alone.
Key concepts: vector database, index, Chroma, persistence and distance defined
| Term | Definition |
|---|---|
| Vector database | A store that keeps embeddings with their text and metadata and finds the items whose numbers are closest to a query’s numbers. |
| Index | The searchable copy of your documents inside the store. Here it is the policy_index/ folder; it goes out of date if the policy files change and it is not rebuilt. |
| Chroma | An open-source database for embeddings that can run locally, be self-hosted or be used as Chroma Cloud. Here it runs from a folder on your computer. |
persist_directory |
The Chroma setting that makes it write to a folder on disk; without it the index lives only in memory and is gone when the script ends. |
Chroma.from_documents |
The call in build_index.py that embeds every chunk and stores its text, numbers and metadata together. |
similarity_search_with_score |
The call in search_index.py that returns the TOP closest chunks, each with a distance. |
| Distance | A number for how far a chunk is from the question: smaller is closer. It is the other way round from session 13’s similarity score. |
| FAISS | A library for efficient similarity search and clustering of dense vectors, written in C++ with Python wrappers. |
| pgvector | An open-source extension that adds vector similarity search to Postgres. |
| Pinecone | A managed vector database service built for AI applications. |
Explanations: how do you save embeddings and search them?
Why not embed every chunk each time the program runs?
Think of a student who copies each textbook by hand at the start of every exam, and throws the copies away afterwards. That is what closest_chunk.py does with the chunks: every run embeds all 34 again and keeps nothing. With 34 chunks that is fine. With many more, starts get slower, and the work repeats for no reason, because the policies have not changed.
The fix is to embed once and save the numbers. A cook who chops vegetables once and keeps them in the fridge does the same. A vector database keeps each chunk’s text, its numbers and its metadata, saved to disk, and finds the closest ones to a question.
What is a vector database, and what is Chroma?
A filing cabinet with labelled drawers is one picture: each folder holds a document and a label, and a query points to the nearest folders. A library catalogue is another: you search the catalogue, and it points you to the shelf. In our design the work splits in two:
build_index.py (run once, and after a policy changes)
load_chunks() -> embed all 34 -> save in policy_index/
search_index.py (run for every question)
open policy_index/
embed the QUESTION only -> closest chunks + distance
The database stores and searches; the embedding model only makes the numbers. Chroma’s own introduction describes it as open-source data infrastructure for AI under the Apache 2.0 licence, which can run locally, be self-hosted or be used as Chroma Cloud. A notebook kept on your desk is a fair picture of the local mode: close the program, open it tomorrow and the notebook is still there.
How does build_index.py save the chunks?
Run python build_index.py once. The core of it is two lines:
chunks = load_chunks()
index = Chroma.from_documents(
chunks, embeddings, persist_directory=str(INDEX_FOLDER))
Chroma.from_documents embeds every chunk with the embedding model and stores text, numbers and metadata together. persist_directory is what writes it to disk. The Chroma class comes from the langchain-chroma package, which takes a persist_directory argument to save data across runs. Ours printed:
Saved 34 chunks to policy_index/
A new folder policy_index/ now exists next to the scripts. It holds each chunk’s text, its numbers and its metadata.
Failure first. HR changes the learning budget in learning-policy.md from 600 to 900, and the helper still says 600. The cause is that the index is a copy of your files: when a file changes, the copy stays as it was. The code handles this the simplest way for a small set of files. Step 3 of the script throws the copy away and builds it again:
if INDEX_FOLDER.exists():
shutil.rmtree(INDEX_FOLDER)
So run build_index.py whenever a policy changes. A second run removes the folder and builds it fresh, so it does not leave two copies of every chunk.
How does search_index.py find the closest chunks, and what is a distance?
The search script opens the saved folder and asks Chroma for the closest chunks:
index = Chroma(persist_directory=str(INDEX_FOLDER),
embedding_function=embeddings)
for chunk, distance in index.similarity_search_with_score(
question, k=TOP):
print(f"distance {distance:.3f}")
Nothing is embedded for the chunks here: the numbers are read from disk. The embedding model is still needed so the question can be embedded the same way. This is the session 13 rule again, with the same failure: if you build with one embedding model and search with another, the results are meaningless. Run python search_index.py. Ours printed (distances depend on the embedding model):
Question: Can I keep leftover holiday days for next year?
distance 0.583 leave-policy.md / 4. Annual leave
4.1 Request annual leave in the HR portal before you book travel. 4.2 Up to 5 unused annual-leave days carry over into the next leave year. Carried-over days must be used by 31 March; days not used
distance 0.634 leave-policy.md / 4. Annual leave
4.6 Leave you have already taken cannot be swapped for pay. 4.7 Two people in the same team may not take the same week of leave unless the manager agrees. 4.8 Employees who leave the company are p
distance 0.665 leave-policy.md / 2. Leave year
2.1 The leave year runs from 1 April to 31 March. 2.2 Each employee's balance is shown in the HR portal and is updated at the start of every month.
It is the same top three as session 13, found without embedding any chunk. The number is a distance, so smaller is closer, the other way round from session 13’s similarity score, where bigger was closer. A distance, like a score, has no pass mark; read the order and the text. If the folder is missing, the script stops with “No index yet. Run: python build_index.py”. Ask your own question: python search_index.py "Who pays for my online course?".
FAISS, pgvector and Pinecone: where does each run, and when would you choose each?
Choosing a store is like choosing where to keep the company’s files: in a desk drawer, in the accounting system you already have, or with an outside archive firm. The four options differ mostly in where they live. This part is theory: the code in this session uses Chroma only.
| Option | What it is | Where it runs | Choose it when |
|---|---|---|---|
| Chroma | Open-source embedding database | Locally, self-hosted or Chroma Cloud | Learning and small projects |
| FAISS | A library for similarity search of dense vectors | Inside your Python program | You want fast search and manage storage yourself |
| pgvector | An open-source extension for Postgres | Inside Postgres | Your data is already in Postgres |
| Pinecone | A managed vector database service | On the vendor’s service | You prefer a managed service |
The FAISS project describes itself as a library for efficient similarity search and clustering of dense vectors, written in C++ with complete wrappers for Python, with some algorithms available on a GPU. Because it is a library, the storage around it is your code’s job. The pgvector project describes itself as open-source vector similarity search for Postgres, and says you keep Postgres features such as ACID compliance, point-in-time recovery and joins. Pinecone describes itself as a fully managed vector database built for AI. The choice follows where your data and your team already are.
What are three situations where the index goes wrong, and what does each mean for the design?
- The helper still gives the old figure after HR changed a policy file. The index is a copy, and it was not rebuilt.
No index yet. Run: python build_index.py.search_index.pywas started before the folder existed, or from another folder.- Search results look random after a change. The index was built with a different embedding model, or the policies changed, so run
build_index.pyagain.
| What you learned | The design decision | Where it shows |
|---|---|---|
| Embedding every run repeats work | Build the index once and save it | build_index.py |
| An index is a copy | Rebuild it whenever a policy changes | The shutil.rmtree step |
| Build and search must share a model | One EMBED_MODEL for both scripts |
The settings at the top of each |
| Stores differ in where they live | Choose by where your data already is | The comparison table |
Basic assignment: edit a line in data/policies/leave-policy.md (change the 5 carry-over days to 8), search again without rebuilding, then rebuild and search again. When does the new number appear, and why?
Project step: build and search the index. Run python build_index.py, then python search_index.py, and check that the same top three come back as in session 13. Change TOP to 1, then to 8, and see how far down the list the first clearly off-target chunk is. Delete the policy_index/ folder and run search_index.py to see its message. The build prompts are prompts/1-write-build-index.md and prompts/2-write-search-index.md. Stop when you have built the folder, searched it and rebuilt it after an edit.
Practice: explain and diagnose. First, answer the Model answer question below aloud in three beats before reading it. Second, for each situation, decide whether the fix is to rebuild the index, to check the embedding model, or to change nothing:
- The helper quotes a figure you edited a minute ago.
- Distances look meaningless after you changed
EMBED_MODELin one script. - A second
build_index.pyrun printed “Saved 34 chunks” again.
Flashcards: can you answer these vector database questions aloud?
Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.
1. What does a vector database do that session 13's list of numbers did not?
It saves the numbers with their text and metadata, and finds the closest ones to a question. The list in session 13 was rebuilt on every run and gone when the script ended.2. What is Chroma?
An open-source database for embeddings that can run locally, be self-hosted or be used as Chroma Cloud. Here it runs from a folder on your computer.3. How do you make Chroma save to disk?
Pass `persist_directory` when you build the index. Without it, the index lives only in memory and is gone when the script ends.4. What does `Chroma.from_documents` do?
It embeds every chunk with the embedding model and stores text, numbers and metadata together.5. What does `search_index.py` embed?
Only the question. The chunks' numbers are read from the saved folder.6. What does a distance of 0.583 against 0.665 tell you?
The chunk with 0.583 is closer to the question. A smaller distance means a closer chunk, the other way round from session 13's similarity score.7. Why must the build and the search use the same embedding model?
The saved numbers only match numbers from the model that made them. Mixing models makes the results meaningless.8. Why does the helper keep quoting an old policy after an edit?
The index is a copy of the policy files, so it stays as it was until you run `build_index.py` again.9. Why does `build_index.py` remove the folder before building?
An old folder would keep chunks of files you have since changed or deleted. Removing it builds the index fresh from the files as they are now.10. How do FAISS, pgvector and Pinecone differ from Chroma?
They differ mostly in where they run. FAISS is a library inside your program, pgvector is an extension inside Postgres, and Pinecone is a managed service. The code in this session uses only Chroma.Model answer: how do you keep the helper’s search quick and its index current?
The core question this session answers: “A colleague asks: why does the helper not embed the policies every time it starts, and what do I need to do when a policy file changes?”
Three beats, in this order: save once · open and search · rebuild when files change. Skip any of these three and the answer is incomplete.
- Save once:
build_index.pyembeds every chunk and saves text, numbers and metadata in a Chroma folder, so the work is not repeated. - Open and search:
search_index.pyopens the folder and embeds only the question, with the same embedding model; a smaller distance means a closer chunk. - Rebuild when files change: the index is a copy, so after a policy edit run
build_index.pyagain.
The full answer, said aloud:
(Save once) “Embedding every chunk on every start repeats the same work. So
build_index.pyembeds the 34 chunks once and saves their text, numbers and metadata in a Chroma folder.(Open and search) Then
search_index.pyopens that folder and embeds only the question, using the same embedding model that built it. Chroma returns the closest chunks with a distance, and a smaller distance means closer.(Rebuild when files change) The index is a copy of the policy files, so when a policy changes I run
build_index.pyagain. Until then the helper answers from the old text.“
Missing a beat is a failure state. The follow-up question lands where you went quiet: skip the rebuild and the next question is why the helper quotes an old figure. Practise until the three come out without prompting.
Why this matters: where do vector databases matter at work?
In a design review: a team is asked why a knowledge assistant starts slowly, or why it quotes a policy that was changed. The answers are in how the index is built and when it is rebuilt.
In a tooling choice: which store to use depends on where the data and the team already are. A team that already runs Postgres looks at pgvector; a team that would rather not run a database looks at a managed service.
In the rest of this course: session 15 opens this same folder and answers from the chunks it finds, and session 17 tunes how many chunks come back.
After this unit: what is the assignment and what comes next?
Assignment. Edit one policy line, search without rebuilding, then rebuild and search again. The brief for the project work is the project file. Keep policy_index/ and your scripts: session 15 opens the folder.
What’s next. Session 15 covers RAG: the helper answers from the retrieved chunks instead of a whole policy file, and cites the source of the chunk it used.
Want help building this with someone?
Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.
Ask about mentor-led trainingTechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.