Courses / Agentic AI from Scratch (Python)
Session 34 of 40
Testing an AI app: test sets, a model that grades, catching hallucinations, measuring an agent
Notes
How to Test an LLM App: Test Sets, LLM-as-Judge, Hallucination Detection and Agent Metrics
Agentic AI Foundations · Session 34
Before you start: what do I need before I test an LLM app automatically?
You need the code folder for this session open in your terminal. It holds the multi-agent Copilot as you left it after session 33, plus five new scripts: code/grader.py, code/grade_five.py, code/check_support.py, code/check_agent.py and code/score_test_set.py. It also holds two data files, code/data/test_set.json with ten test questions and code/data/hand_graded.json with five answers a person has already marked. Install the libraries with pip install mcp langchain langchain-openai langchain-chroma langchain-community langchain-text-splitters langfuse, set LLM_BASE_URL, LLM_API_KEY, LLM_MODEL and EMBED_MODEL as in earlier sessions, then run python code/build_index.py and python code/make_tickets_db.py. The Langfuse keys are optional: without them the scripts run untraced. Have paper or a notes app ready for the flashcards, the model answer and your notes/scores.md.
Summary: how do I score an AI app automatically and know when the score is wrong?
A pass count made by hand does not scale: nobody re-reads ten answers after each small change. The answer is a test set that runs on its own, with a second model, called a judge or grader, marking each answer against a reference answer a person wrote. That gives the Copilot one score to compare after each change. A grader is itself a model, so it needs checking against a person, and it can miss what it was not shown. This session adds two more checks: whether each claim in an answer is backed by the text the Copilot was given, which catches made-up answers, and whether the supervisor used the right agents and tools. The project step is the Copilot’s first automatic score.
Key concepts: what do the AI testing terms mean?
| Term | Definition |
|---|---|
| Test set | A fixed list of questions, each with the answer you expect, run again each time something changes. |
| LLM-as-judge (grader) | A second model call that reads the question, a reference answer and the Copilot’s answer and returns a score from 1 to 5 with a reason. |
| Pass score | The lowest judge score that counts as a pass; in code/grader.py it is 4. |
| Judge score | The count of test answers that passed, with their average, printed by code/score_test_set.py so runs can be compared. |
| Reference-guided grading | Grading where the judge is given a reference answer to use as the benchmark. |
| Agreement | How often the grader’s marks match a person’s marks on the same answers. |
| Hallucination | A fluent answer that is not supported by the sources, or by fact. |
| Groundedness (faithfulness) | How far the claims in an answer are supported by the retrieved text. |
| Claim | One checkable statement inside an answer, such as one figure, rule or condition. |
| Agent check | A check of what an agent did: which agents and tools ran, and whether the goal was met. |
| Goal check | Checking the outcome of a task, not the steps taken to reach it. |
Explanations: how does automatic AI testing work?
How does LLM-as-judge grading work?
Think of a teacher with a mark scheme. The scheme says what a good answer contains, and the teacher applies it to each script. A driving examiner works the same way: one checklist for each candidate and a pass or fail at the end. An LLM-as-judge is that role in code. OpenAI’s evaluation guidance observes that a model judge costs less to run and scales better than human review, and it describes two styles: reference-guided grading, where the judge is given a reference answer as the benchmark, and single answer grading, where the judge looks at one response on its own. The Copilot’s grader is reference-guided, because each test question has a reference answer.
class Grade(BaseModel):
score: int = Field(description="1 to 5 ...")
reason: str = Field(description="One short reason ...")
def grade(question, reference, answer):
result = judge.invoke({"question": question,
"reference": reference, "answer": answer})
result.score = max(1, min(5, result.score))
return result
PASS_SCORE = 4 # a score of 4 or 5 counts as a pass
This is the heart of code/grader.py. The judge returns a typed reply, as in earlier sessions on structured output: a score from 1 to 5 and one short reason. Its rules are in code/model_prompts/judge.md: a wrong figure or a wrong condition scores 2 or lower however fluent the answer sounds, and length, politeness and confident wording earn nothing. The same OpenAI guidance recommends a pass or fail threshold beside a numerical score, so you decide in advance what counts as good enough; here that line is PASS_SCORE.
Why not match the text word for word? Because “five” and “5” are the same answer, and a word-for-word match would fail a right reply. A grader reads meaning. The same file also holds two checks in plain code, cites_a_policy and contains, which need no model and cannot be charmed by fluent wording. The guidance asks for typical cases, edge cases and adversarial cases in test data, and it notes that models sometimes produce different output from the same input, so read a score as a guide rather than an exact figure.
What does the test set hold?
code/data/test_set.json has ten questions. Five are policy questions: the carry-over of unused leave, what to attach to an expense claim, how many days a week you can work from home, the yearly learning budget, and the home-office chair allowance. Two are ticket questions: the status of T-1043 and who is working on T-1046. One is the mixed Wi-Fi question that needs a policy answer and a ticket. Two have no answer in the files on purpose: what the travel policy says about booking flights, where the right reply is that there is no such policy and a person should take over, and the weather in Paris, which is not a help-desk question. Each entry carries a reference answer, a key word the reply should contain, and the agents and tools the supervisor is expected to use.
When is the LLM grader wrong, and how do I check it?
Show the failure first. code/data/hand_graded.json holds five answers that a person has already marked: three pass and two fail. One failing answer states the wrong figure, ten days of carry-over where the reference says five. The other states a rule that sounds natural but is wrong: that the learning budget carries over, when the reference says unused budget does not. A judge that is too generous would pass both, because both read well.
Run python code/grade_five.py. It grades the same five answers and, for each, prints the person’s mark, the judge’s verdict and score, whether they agree, and the judge’s reason. The last line counts the agreements. Look at any disagreement and read its reason. Two examiners can mark the same script differently, and a model judge has habits too: OpenAI’s guidance names position bias, which is a leaning that depends on response order, and verbosity bias, which is a preference for longer responses. It says to validate agreement against your human labels before relying on the judge. A judge that disagrees with a person is the thing you fix, not the person: open code/model_prompts/judge.md, change one rule, run the script again, and see whether the count moves.
How do I detect hallucinations in an LLM answer?
A fact-checker with the original file matches each sentence in an article to a page. A marker with the textbook open accepts an answer when the book backs it. A hallucination check works like that. The Ragas faithfulness metric states the idea: a response is faithful if its claims can be supported by the retrieved context. To measure it you identify the claims in the response, check each against the retrieved context, and compute the share that is supported.
Now the failure first. In code/check_support.py, Part A checks two made-up answers against the leave policy. The first restates the carry-over rule. The second restates it and adds a payment for each day lost, which the policy does not mention. The second sounds sure and reads as plausible, yet nothing backs the extra claim. The cause is that a model writes what a policy would plausibly say.
class Support(BaseModel):
supported: bool
unsupported_claims: list[str]
def check_support(answer, context):
return checker.invoke(
{"context": context, "answer": answer})
The key difference from the plain grader is what the judge sees: the policy text, not just a reference answer. A grader that was not shown the text has no way to tell that a claim was invented. The rules are in code/model_prompts/support.md: each figure, rule or condition is one claim, and a claim is supported only when the policy text says it, even if it sounds sensible and is true in the real world. Part B of the script asks the real helper one question and checks its answer against the pieces the search returned. Pair this check with the plain code check: a cited policy must be present.
How do I measure whether an agent did the right thing?
Think of a driving test. The examiner checks that you took the right turns, and also that you arrived. An agent can be measured the same way: its steps and its outcome. Ragas, an evaluation library, separates the two in its agent metrics. Its tool-call accuracy fits when you care about the exact tool sequence and arguments, and its agent goal accuracy fits when you care about the outcome and not the specific tools used.
In the Copilot, multi_agent.py writes a line to LOG whenever an agent or tool runs, and keeps the policy text the search returned in CONTEXTS. ask() starts each question with an empty LOG. code/check_agent.py reads those lines into two sets, the agents used and the tools used, and check_agents() compares them with the ones in the test case. Both must match exactly, so a missing agent and an extra one are both wrong. A separate goal check, no_ticket_written(), confirms that no ticket was created after the person said no. Run python code/check_agent.py: it asks four questions, a policy question, a ticket question, the mixed Wi-Fi question and one that is not about the help desk, and prints the expected and the used agents and tools for each. A new ticket is refused automatically, so nothing is written.
Keep the checks apart. An agent can pick the right tool and still fail the task, and it can reach a correct answer through a wrong tool by luck. A strict tool check is right for a fixed route and too strict for a task with several valid routes; for those, check the goal.
How do I score the whole test set?
Run python code/score_test_set.py. For each of the ten questions it asks the Copilot, grades the reply with the judge, checks the key word in plain code (and a named policy for a policy answer), checks that a policy answer is backed by the text the search returned, and checks the agents and tools. It prints one line per question, and the judge’s reason under any answer that failed. The last lines give the judge score, how many of the ten passed with their average out of five, and then three other counts: the code check, the backed-by-the-text count and the agent check.
The script prints and saves nothing, so copy the numbers into notes/scores.md with the failed questions and the reasons. A score is a comparison, not a verdict: a model’s choices vary, so a question that fails once is worth running a second time before you change anything. After each change to a prompt or a script, run it again and compare the averages, not single questions.
What goes wrong when you test an AI app badly?
- A prompt change feels better, and test questions fail later. Nobody scored the test set before and after, so there was no number to compare. Fix: score each change.
- The judge passes an answer with a wrong rule that sounds natural. Nobody compared the judge with a person. Fix: run
code/grade_five.pyand tightenmodel_prompts/judge.md. - An answer adds an invented payment and the judge passes it. The judge was not shown the policy text. Fix: give the support check the text the Copilot was given.
| What you learned | The design decision | Where it shows |
|---|---|---|
| A grader is another model | Check it against a person; add code checks | code/grader.py, code/grade_five.py |
| A score is a comparison | Score the test set after each change | code/score_test_set.py |
| Fluent is not supported | Check each claim against the text | code/check_support.py |
| Steps and outcome differ | Check the agents, tools and goal separately | code/check_agent.py |
Flashcards: test yourself on AI testing
Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.
1. What is an LLM-as-judge?
A second model call that reads the question, a reference answer and the Copilot's answer and returns a score from 1 to 5 with a reason. Like a teacher with a mark scheme, it applies the same standard to each answer.2. What is reference-guided grading?
Grading where the judge is given a reference answer to use as the benchmark. The Copilot's grader works this way because each test question has a reference answer.3. Why not compare answers word for word?
Because "five" and "5" mean the same thing, and a word-for-word match would fail a right answer. A grader reads meaning instead, while small plain-code checks cover the key word.4. What are two known problems with a model judge?
Position bias, where the order of responses affects the verdict, and verbosity bias, a preference for longer responses. Validate agreement against a person's marks before relying on the judge.5. How do you check a grader before trusting it?
Run it on answers a person has already marked, as `code/grade_five.py` does, count how often it agrees, and read the reasons for the disagreements. Then fix the judge's rules, not the person's marks.6. What does a hallucination check test?
Whether each claim in an answer is supported by the text the Copilot was given: split the answer into claims, check each against the text, and list the unsupported ones. A fluent answer can still be unsupported.7. What must the support check see that the plain grader may not?
The policy text. A grader that was not shown the text has no way to tell that a claim, such as an invented payment, was made up.8. What does the agent check compare?
The agents and tools recorded in `LOG` with the ones the test case expects. Both sets must match exactly, so a missing agent and an extra one both fail.9. When do you check the goal and not the tool order?
When a task has several valid routes. A strict tool check suits a fixed route; for an open task, check the outcome instead, as `no_ticket_written()` checks that nothing was created after a no.10. Why score the test set after each change?
A score is a comparison: a drop after a change points at the change. Without a before-and-after number, a change that hurt goes unnoticed until a user complains.Model answer: how do I explain LLM evaluation in an interview?
The core question this session answers: “A teammate asks: how do we know the Copilot still answers well after a change, how far can we trust a model that grades, and how do we catch made-up answers and bad tool choices?”
Four beats a teammate listens for, in this order: score the test set · check the grader against a person · check claims against the text · check the tools and the outcome. Skip any of these four and the answer is incomplete.
- Score the test set: a grader marks each question and gives one score.
- Check the grader against a person: count agreements; add code checks for gaps.
- Check claims against the text: a support check on the text the Copilot was given.
- Check the tools and the outcome: separate checks for an agent.
The full answer, said aloud:
(Score the test set) “I run the test set through a grader model that marks each answer against a reference, and I keep the score. After each change I run it again and compare.”
(Check the grader against a person) “A grader is another model, so I run it on answers a person has marked and count agreements. Where it misses something, I fix its rules and add a plain code check.”
(Check claims against the text) “To catch made-up answers I list each claim in the answer and check it against the text the Copilot was given. A fluent answer with an invented claim is flagged.”
(Check the tools and the outcome) “For the agent I compare the agents and tools it used with the expected ones, and I check separately that the goal was met, because it can do one without the other.”
Missing a beat is a failure state. The follow-up question lands where you went quiet: skip the last beat and you are asked how you know the agent picked the right tool. Practise until the four come out without prompting, then stop talking and let the questions come.
Why this matters: why does automatic testing matter at work?
In a conversation with a teammate or manager: a product owner approving a release asks whether the last change made answers better or worse. A judge score from before and after, with the failures listed, is an answer; “it seems better” is not.
On the job: a score that runs after each change catches regressions quickly instead of in a customer’s complaint. Knowing how far to trust a grader keeps the number honest.
In the rest of this course: session 35 adds attacks on the Copilot, such as prompt injection, that you run as checks, and session 36 uses what you can measure as evidence in the risk register. The capstone’s evaluation work builds on these scores.
After this unit: what do I do after this lesson, and what comes next?
Assignment. Run python code/grade_five.py and count the agreements, then run python code/check_support.py and read the claim it flags as unsupported. Project step. Write code/grader.py, code/grade_five.py, code/check_support.py, code/check_agent.py and code/score_test_set.py (the build prompts are code/prompts/1-write-grader.md to code/prompts/5-write-score-test-set.md), score the Copilot on the ten test questions, and keep the score and the failed questions with their reasons in notes/scores.md. The brief for the project work is the project file. Next session: security, starting with prompt injection.
Want help building this with someone?
Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.
Ask about mentor-led trainingTechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.