Courses / Agentic AI from Scratch (Python)

Session 38 of 40

Capstone build 1: pick a domain, design the solution, start building

Notes

How to Set Up an LLM Capstone: Data Packs, One Setting and a Pack Check

Agentic AI Foundations · Session 38

Before you start: what do I need before I run my Copilot on a data pack?

You need the code folder for this session open in your terminal, and every command below runs from inside it, as the folder’s README.md does. It holds the Copilot, its API and its chat page from session 37, plus two new scripts, code/pack.py and code/check_pack.py, and a new place for data, code/data/packs/, with four packs: helpdesk, banking, healthcare and procurement. The libraries are the ones you already have, so there is nothing new to install. Set LLM_BASE_URL, LLM_API_KEY, LLM_MODEL and EMBED_MODEL in your terminal as in earlier sessions. Have paper or a notes app ready for the flashcards, the model answer and your one-page design sheet. Every organisation in the packs is invented; use invented names in your own pack too.

Summary: how do I reuse one AI assistant for different organisations?

The program you built does not know what a leave policy or a card dispute is. The knowledge sits in documents, records and test questions, so a new organisation is a new folder of data and not new code. A data pack is that folder. One setting in your terminal, DATA_PACK, chooses which pack the Copilot runs on, and only code/pack.py reads it. A second script, code/check_pack.py, checks that a pack is complete before you build on it. Today you switch between packs, test that the role rule still holds on each, and start a pack and a set of test questions of your own for the capstone.

Key concepts: what do the terms for data packs mean?

Term Definition
Data pack A folder holding one organisation’s documents, records, test questions and roles.
pack.json The small file that names the company, its topics, its roles and its restricted files.
DATA_PACK The one setting, in your terminal, that chooses the pack.
Environment variable A value that lives outside the Python code, in the operating system, and the program can read it.
Restricted file A policy file that only one role may read.
Default role The first role listed in pack.json, the one with the fewest rights.
Test set The pack’s test_set.json: five to eight questions, each with the answer a person expects.
Adversarial case A test question that tries to get something the caller should not have.
Design sheet One page: three questions, the files, the action that needs a yes, one role rule, two risks and how you will test.

Explanations: how does one Copilot run on four organisations?

What is a data pack, and what does it hold?

Think of a school timetable swapped for another school’s: the clock and the bells stay the same, and only the lessons differ. Each folder in code/data/packs/ holds four things.

Part What it is
policies/ Three to five short documents. One is for a single role only. One is a notes page with an instruction planted in it, text an attacker could have edited in, as in session 35.
tickets.csv The records, with the same four columns in every pack: id, title, status and assignee.
test_set.json Five to eight test questions, each with the answer a person expects.
pack.json The company, the topics, the roles and the restricted files.

The four packs, all invented organisations:

Pack Organisation Roles Restricted file
helpdesk Fernhill Systems employee, hr salary-bands-policy.md
banking Pellwick Credit Union staff, fraud-team fraud-review-notes.md
healthcare Tarnbrook Clinic front-desk, clinician clinician-notes.md
procurement Wexcombe Supplies staff, purchasing supplier-price-terms.md

The healthcare pack gives no medical advice: its policy says to ask a clinician, and one test question checks it. This is what pack.json looks like for banking:

{
  "company": "Pellwick Credit Union",
  "topics": "account fees, card disputes, ...",
  "roles": {"staff": ["everyone"],
            "fraud-team": ["everyone", "fraud-team"]},
  "restricted_files": {"fraud-review-notes.md": "fraud-team"}
}

The roles map a name to the audiences it may read. A role that is not named may read nothing. The first role listed is the default, and it may not read the restricted file.

How does one setting switch the whole Copilot?

The failure first. If a script opened data/policies/ by name, switching pack would break it, because that folder no longer holds the documents. So one script, code/pack.py, reads the setting, and every other script imports it and asks it for a path.

DATA_PACK = os.environ.get("DATA_PACK", "helpdesk")
PACK_FOLDER = PACKS_FOLDER / DATA_PACK
POLICY_FOLDER = PACK_FOLDER / "policies"
TICKETS_CSV = PACK_FOLDER / "tickets.csv"
TEST_SET = PACK_FOLDER / "test_set.json"
BUILD_FOLDER = HERE / "build" / DATA_PACK
ROLES = INFO["roles"]

An environment variable is a value outside the code, so the same program can run in different places with different settings, and a value read from the environment arrives as text. With no DATA_PACK the pack is helpdesk. A name that is not a pack stops with a plain message that lists the names to choose from, instead of failing later with a path error. These parts also follow the pack:

  • The prompts in code/model_prompts/ have {company} and {topics} blanks that pack.fill fills in, so one prompt serves every pack.
  • The index and the ticket database of each pack are built into build/<pack>/, so switching back needs no rebuild.
  • The search, the audit and the API take their folders and roles from the pack. Each audit record and request-log line says which pack answered, and /health and the chat page name it.
  • The API’s keys must use roles of the chosen pack, such as staff and fraud-team for banking.

Build each pack with the same DATA_PACK you ask with. If the terminal that built the index and the one that asks differ, the Copilot answers with the wrong company’s facts. Run python pack.py in both terminals to see which pack each one will use.

How do I check that a data pack is complete?

code/check_pack.py reads a pack’s folder and checks the parts the Copilot needs: pack.json has company, topics, roles and restricted_files; policies/ holds at least three files; each restricted file is in policies/ and names a role that exists in roles; tickets.csv has the columns id, title, status and assignee, with ids that look like T-1042; and test_set.json holds five to eight questions of a known kind. It reports every problem it finds and exits with 1 if there is one. With no flag it checks the chosen pack, and with --all it checks all four.

A check like this is cheap insurance. A pack with one wrong role name would otherwise fail much later, as a Copilot that says “I don’t know” to everything.

Which test questions does a pack need?

Each question in test_set.json has a kind.

Kind What it tests
policy A typical question answered from the documents
ticket A question about a record
mixed Needs the policy agent, then the ticket agent, and a person’s yes
restricted A question the default role may not read
other An off-topic question

OpenAI’s evals guidance says test data should include typical cases, edge cases and adversarial cases. The kinds above cover all three: typical, the off-topic edge, and the restricted question as an adversarial one. The test set also gives the Copilot its default question for each kind, which is why python ask_copilot.py with no question has something to ask. Session 39 runs the whole set and grades it.

How do I test the role rule on a pack?

The role rule of session 35 carries over to the four packs because code/secure_search.py reads the pack’s roles and its restricted file. Ask the restricted question with each of the pack’s roles. The default role should get an “I don’t know”, and the other role should get an answer with a source. The restricted file never reaches the model for the first role, because the search filters it out before the model reads anything. The role pairs are staff and fraud-team for banking, front-desk and clinician for healthcare, staff and purchasing for procurement, and employee and hr for helpdesk.

What goes wrong when a pack is set up carelessly?

  1. A banking question gets the helpdesk company’s facts. DATA_PACK differed between the terminal that built the index and the one that asked. Fix: run python pack.py in both, and rebuild with the right setting.
  2. Every answer says “I don’t know”. The role is spelled wrongly, and a role that is not in pack.json may read nothing. Fix: use one of the pack’s roles.
  3. The check says a restricted file names a role that is not in roles. A role was renamed in one place and not the other. Fix: make the role in restricted_files one of the keys of roles.
What you learned The design decision Where it shows
The domain is knowledge, not logic Put it in a data pack code/data/packs/
A setting read in many places drifts One script reads it code/pack.py
A broken pack fails late and oddly Check it first code/check_pack.py
A role rule must survive a change of domain Test the restricted question on each pack code/secure_search.py, code/ask_copilot.py

Flashcards: test yourself on data packs

Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.

1. What does a data pack hold? A `policies/` folder of documents, a `tickets.csv` of records, a `test_set.json` of test questions and a `pack.json` that names the company, topics, roles and restricted files.
2. How do you switch the Copilot to another organisation? Change one setting in the terminal, `DATA_PACK`, and build the index and ticket database for that pack. No code changes.
3. Which script reads `DATA_PACK`, and why only one? `code/pack.py`. Every other script asks it for a path, so a switch leaves no script pointing at an old folder.
4. What is the default pack? `helpdesk`, used when `DATA_PACK` is not set.
5. Why do prompts have `{company}` and `{topics}` blanks? `pack.fill` fills them from `pack.json`, so one prompt file serves every pack.
6. Where are the index and ticket database of a pack built? In `build//`, so each pack has its own and switching back needs no rebuild.
7. What does `check_pack.py` check? That `pack.json` has its four keys, `policies/` holds at least three files, each restricted file exists and names a real role, `tickets.csv` has the right columns and ids, and `test_set.json` holds five to eight questions of a known kind.
8. What are the kinds of test question? `policy`, `ticket`, `mixed`, `restricted` and `other`: typical questions, a record question, one that needs a yes, one the default role may not read, and an off-topic one.
9. What should the default role get for the restricted question, and the other role? The default role gets an "I don't know", because the search filtered the restricted file out before the model read anything. The other role gets an answer with a source.
10. Why does a role not named in `pack.json` get "I don't know" for everything? A role that is not in `pack.json` may read nothing, so check the spelling and use one of the pack's roles.

Model answer: how do I explain a reusable AI assistant in an interview?

The core question this session answers: “A manager asks: how did you make the same Copilot work for a bank and a clinic, and how do you know a new one is ready?”

Four beats a manager listens for, in this order: the domain is data · one setting · check the pack · test the role rule. Skip any of these four and the answer is incomplete.

  1. The domain is data: documents, records, test questions and roles.
  2. One setting: DATA_PACK, read in one place.
  3. Check the pack: check_pack.py before anyone builds.
  4. Test the role rule: the restricted question with both roles.

The full answer, said aloud:

(The domain is data) “Everything that is specific to an organisation sits in a pack folder: its documents, its records, its test questions and its roles. The code has no company facts in it.

(One setting) One setting in the terminal chooses the pack, and only one script reads it. Every other script asks that script for its paths and names, so switching changes no code.

(Check the pack) Before I build on a pack, a script checks its files, its ticket columns and the number of test questions, and tells me every problem at once.

(Test the role rule) On each pack I ask the restricted question with both roles. The default role gets an I-don’t-know and the other gets an answer with a source, which shows the access rule survived the change.“

Missing a beat is a failure state. The follow-up lands where you went quiet: skip the last beat and you are asked how you know a new pack does not leak. Practise until the four come out without prompting, then stop talking and let the questions come.

Why this matters: why does a reusable setup matter at work?

In a conversation with a teammate or manager: a second customer is the first real test of a design. Saying that the domain lives in data and that a script checks a new pack shows that you built for reuse.

On the job: a handover goes better when a new person can switch to a known data set, run a check and see the same behaviour. A test set and a role rule tested per pack are how you show the Copilot still behaves.

In the rest of this course: session 39 runs your pack’s test set and attacks to produce an evaluation report and a security report, and session 40 gives the final demo on the pack you choose now.

After this unit: what do I do after this lesson, and what comes next?

Assignment. Run python pack.py and python check_pack.py --all, then build and ask on the helpdesk pack. Switch with export DATA_PACK=banking, build again, and ask python ask_copilot.py "What is the status of case T-2002?". Run the restricted question with both roles of a pack. Then choose your capstone pack, copy a pack folder to code/data/packs/mine, change the company, policies, tickets and test questions to your own invented idea, and run DATA_PACK=mine python check_pack.py until it prints OK. Keep a one-page design sheet: three questions people ask, the files in the pack, the action that needs a person’s yes, one role rule, two risks and how you will test. Project step. Write code/pack.py and code/check_pack.py (the build prompts are code/prompts/1-write-pack.md and code/prompts/2-write-check-pack.md). The brief for the project work is the project file. Next session: the evidence for the capstone documents.

Want help building this with someone?

Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.

Ask about mentor-led training

TechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.