Courses / Agentic AI from Scratch (Python)
Session 8 of 40
Prompt engineering 3: testing prompts against a test set, and a prompt library
Notes
How to Test an LLM Prompt: Breaking Prompts, Test Sets, Pass Counts and a Prompt Library
Agentic AI Foundations · Session 8 — Prompt engineering 3: testing prompts against a test set, and a prompt library
Before you start: what do you need to test prompts and keep a prompt library?
You need the code folder for this session, chapters/08-test-set-and-library/, open in your terminal. It holds helpdesk_agent.py (the session 5 helper with its work wrapped in one function), run_test_set.py, data/test_set.json with nine test cases, and model_prompts/library/ with two prompt versions and a changelog. Set your three settings (LLM_BASE_URL, LLM_API_KEY, LLM_MODEL) as in session 1, either on the top lines of the script or in the terminal. No new library is needed. A test run calls a model at least once per case. Have paper or a notes app ready for the Flashcards and the Model answer.
Summary: how do you know a prompt works, and where do you keep it?
A prompt that gave one good answer has not been shown to work, so this session adds three habits for finding out and one for keeping the result. You break the prompt on purpose with a bad input, such as a ticket or a policy section that does not exist, and write down what the helper does. You run a whole test set and read a pass count, changing one thing between runs so the count tells you what helped, and you run it more than once because a model does not reply the same way every time. Then you keep the prompt out of the code and in a prompt library: named, versioned files and a changelog, so a team can review them and go back to one that worked. The project step scores the helper on both library versions and adds a third.
Key concepts: breaking a prompt, rule, test set, pass count and prompt library defined
| Term | Definition |
|---|---|
| Breaking a prompt | Deliberately sending a prompt an input it was not designed for, to see how it fails before a real user finds out. |
| Rule | A sentence in the prompt that tells the model what to do in a case, such as setting needs_human to true when a ticket is not found. |
| Test set | A fixed list of questions, each with what you expect back, that you run each time the prompt or model changes. |
| Test case | One entry of the test set: a name, a question, whether a person is needed (needs_human) and a word the answer must contain (must_contain). |
| Pass count | How many test cases out of the total got a reply that matches what the case expects. |
ask_helper |
The function in helpdesk_agent.py that takes a question and returns the three reply fields, so a script can call the helper once per case. |
| Prompt library | A folder of named, versioned prompts kept beside the code, with a changelog saying what each version changed. |
| Version | One saved form of a prompt, such as v2; a change is a new file, so the old one can be restored. |
PROMPT_VERSION |
The setting that picks which library file the helper reads, such as system-v2.md. |
Explanations: how do you break, score and store a prompt?
Where are we? What does this session add to the HelpDesk Copilot?
A reply follows five steps, and the last one is the answer itself: “Up to 5 days, used by 31 March”, or “Not in the policy: raise a ticket”. Since session 5 the model picks the policy section itself with a tool call. This session tests step 5: does the answer come out right across many questions, not only the one you tried? The code is chapters/08-test-set-and-library/. helpdesk_agent.py is the helper from session 5 with one new setting and one function, run_test_set.py runs the test set through it, and model_prompts/library/ holds the prompt versions.
What does it mean to break a prompt, and which bad inputs should I try?
An employee asks: “What does Leave Policy 4.9 say?” The leave policy has no section 4.9. Here is what a failing prompt can return:
Leave Policy 4.9 says that employees may take up to 3 extra days of leave for moving house, with a manager’s approval.
That reply is invented, and it was illustrative here, but it is the shape of the failure: a confident answer about a section that does not exist. The cause is in the prompt. It told the helper to name the section it used, but it left out what to do when the section is not there. When a prompt is silent about a case, the model fills the gap with something plausible, because it writes what is likely, not what has been checked.
Breaking a prompt means sending it an input it was not designed for on purpose, to see how it fails before a real user does. A car maker crash-tests a car in a laboratory so the failure happens there; breaking a prompt is the same idea. It is also like pressing the buttons in a new lift before visitors arrive, so the odd one that does nothing is found first.
Bad inputs worth trying on the helper, and what it should do with each:
| Bad input | What can go wrong | What the helper should do |
|---|---|---|
| A ticket that does not exist (T-9999) | An invented status | Say so, and set needs_human to true |
| A section that does not exist (section 9 of the leave policy) | An invented rule | Say so, and set needs_human to true |
| A policy that does not exist (the travel policy) | A guess from general knowledge | Say so, and set needs_human to true |
| A question outside help desk topics (the capital of France) | A chatty answer | Set needs_human to true |
The fix is a rule: a sentence in the prompt for the case. In model_prompts/library/system-v2.md it reads, in part, “Set needs_human to true, and say in answer what is missing, in each of these cases”, followed by four cases: a tool returns “ticket not found” or “no such policy”, the section asked about is not in the policy text, the policy does not answer the question, and the question is not about Fernhill help desk topics. After the fix, ask the same bad question again to see that it now behaves sensibly.
How do I test a prompt against a test set and compare versions with a pass count?
A cook adjusts a recipe. They add more salt and swap the flour in one go, and it tastes better, but they cannot say which change did it. A cook who tasted each version with the same panel, and wrote down scores, would know. A prompt is the same: without a fixed set of questions to check against, a change that seems better is a guess.
A test set is a fixed list of questions, each with what you expect back, that you run each time the prompt or model changes. Think of an answer sheet used for marking: the same questions each time, so scores can be compared fairly. For the helper it is data/test_set.json, nine cases: normal questions (carry-over days, a ticket status, who approves leave, a lost laptop) and bad inputs (a missing ticket, a missing section, a missing policy, a sick leave question the policy does not answer, an off-topic question). One case looks like this:
{"name": "missing ticket",
"question": "What is the status of ticket T-9999?",
"needs_human": true,
"must_contain": ""}
run_test_set.py loads the cases, runs each question through ask_helper, and checks the reply. A case passes when the reply is valid JSON, needs_human matches the case, and the answer contains the word the case expects (if it expects one). These are the three checks, with their messages shown as comments:
if reply is None:
... # "reply was not valid JSON"
elif reply["needs_human"] != case["needs_human"]:
... # "needs_human was True, expected False"
elif case["must_contain"].lower() not in reply["answer"].lower():
... # "answer does not contain '...'"
else:
... # the case passes
The loop around them ends with the pass count, the number to compare:
passed = 0
for case in cases:
try:
reply = ask_helper(case["question"])
except APIError as error:
print("FAIL", "-", case["name"], "-", "the model could not answer:", error.message)
continue
...
passed += ok
print("PASS" if ok else "FAIL", "-", case["name"], "-", why)
print(f"\nPassed {passed} of {len(cases)}")
If the model cannot be reached at all (a wrong key, URL or model name, or no network), ask_helper raises APIError. The runner marks that case FAIL with the reason and carries on with the next one, so one bad call does not stop the whole run.
ask_helper(question) is the first function this course writes besides the two tools. It wraps the whole helper, the tool loop included, so run_test_set.py can call it once per case, which a script that only runs top to bottom cannot offer. The last lines of helpdesk_agent.py start with if __name__ == "__main__":; they run only when you start that file yourself, not when run_test_set.py imports it.
Run it with python run_test_set.py. One real run printed this:
Prompt version: v2
PASS - carry-over days -
PASS - ticket status -
PASS - who approves leave -
FAIL - lost laptop - needs_human was True, expected False
PASS - missing ticket -
PASS - section that does not exist -
PASS - policy that does not exist -
PASS - sick leave allowance -
PASS - off-topic question -
Passed 8 of 9
Now run it two or three times. The model can reply differently from one run to the next, so a case can pass on one run and fail on the next. Think of a cook who bakes the same cake twice, or a student who sits the same quiz twice: most results match and a borderline one flips. In our runs two runs passed 9 of 9, and one failed “lost laptop” with needs_human true. OpenAI’s guidance on evals says the same: models sometimes produce different output from the same input. A case that fails now and then is exactly what a test set is for: it shows where the prompt is weak. Read a count as a guide, not an exact figure. The same guidance says test data should include typical cases, edge cases and adversarial cases, so add the bad inputs you find to your set.
Now the habit that makes the count useful: change one thing between runs. Picture a bicycle that wobbles. You replace the tyres and the seat together, and the wobble stops, but which fix was needed? A garden is the same: new soil, new seed and more water at once, and the plants grow, but you learned nothing. Two changes at once hide which one helped, and one of them may have hurt. Change one thing in the prompt, run the set more than once, record the count, then change the next thing. OpenAI’s guidance also recommends pinning a production application to a specific model snapshot so behaviour stays consistent; a model change is also a change to run the set against.
What is a prompt library, and how do I keep named, versioned prompts beside the code?
Picture a recipe box with dated cards. Each card has a name, a version and a note of how the dish turned out. When someone changes the salt, they write a new card and keep the old one, so you can tell which card produced which dish and go back if the new one was worse. A prompt library is that box for prompts.
Failure first. Two teammates have been editing the helper’s prompt in a chat window and pasting it into the code. The earlier answers were good; the new ones are not, and nobody can say which wording was live before. The prompt was not kept as a file, so there is no history, no review and no way to restore it.
OpenAI’s own prompt-engineering guide has moved in this direction. For new work it advises keeping prompt builders in a small module near the feature they support, and adding representative fixtures, tests and evaluation checks before changing production prompts. In other words, the prompt lives with the code, and a change has to earn its place with a test.
The library for the helper is a folder of plain files:
model_prompts/library/
system-v1.md the first working prompt
system-v2.md + the cases where needs_human must be true
CHANGELOG.md one line per version: what changed and why
The two versions differ in one thing you can name:
| Version | What the prompt says | What changed |
|---|---|---|
system-v1.md |
Names the tools and the three reply fields | The first working prompt |
system-v2.md |
Also lists the cases where needs_human must be true, and asks for figures as digits |
Added after the test set failed on bad inputs |
The helper picks a version with one setting at the top of helpdesk_agent.py, which an environment variable can override:
PROMPT_VERSION = "v2"
...
version = os.environ.get("PROMPT_VERSION", PROMPT_VERSION)
system_message = (LIBRARY / f"system-{version}.md").read_text()
To run the same test set with the older prompt, set the variable for one run:
PROMPT_VERSION=v1 python run_test_set.py
On Windows PowerShell, use $env:PROMPT_VERSION = "v1"; python run_test_set.py. The code is the same and only the prompt file differs, so a different pass count belongs to the prompt. Run each version more than once, since a single run can be lucky, and keep the version with the higher count. A new version is a copy with one change: copy system-v2.md to system-v3.md, change one rule, set PROMPT_VERSION to "v3", and write one line in CHANGELOG.md. A version without a recorded result is a prompt nobody has measured.
What are three situations where a prompt lets you down, and what does each mean for the design?
- After a tweak the prompt “seems better”, but two earlier questions now fail. No test set was run, so nobody noticed that the change broke other answers.
- Two teammates edit the prompt in a chat window and nobody knows which wording was live earlier. The prompt was not kept as a named, versioned file.
- The helper invents a section 4.9. The prompt was silent about a missing section, and nobody had tried that input.
| What you learned | The design decision | Where it shows |
|---|---|---|
| Odd inputs reveal gaps | Add a rule in the prompt for each bad input | system-v2.md |
| A change is a guess until measured | Score each version on the test set | run_test_set.py |
| Two changes at once hide the cause | Change one thing, re-run, record | Your pass counts |
| One run can be lucky or unlucky | Run the set two or three times | The same case failing now and then |
| A prompt in a chat window has no history | Save it as a named, versioned file | model_prompts/library/ |
Basic assignment: break the helper with one bad input, such as a section that does not exist, and write down what it did. Then add one rule to a copy of the prompt, ask the same question again, and check you broke nothing.
Project step: score the helper and keep a new version. Run python run_test_set.py two or three times on v2 and write down each pass count. Run it on v1 the same way and compare. Then copy system-v2.md to system-v3.md, change one rule, set PROMPT_VERSION = "v3", run the set, and write one line about v3 in CHANGELOG.md. The build prompt for run_test_set.py is prompts/2-write-run-test-set.md. Stop when the library holds three versions and you have counts for each. You can also add a tenth case to data/test_set.json that you think will break the helper.
Practice: explain and sort. First, answer the Model answer question below aloud in four beats before reading it. Second, for each of these four situations, decide whether the fix is a rule, a test run, a repeat run or a library file, and write your reason in one sentence:
- A reply invents a section that does not exist.
- Nobody knows whether the earlier prompt was better.
- A case passed yesterday and failed today with the same prompt.
- Nobody can find the earlier wording.
Flashcards: prompt testing, pass count and prompt library questions with answers
Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.
1. What does breaking a prompt mean, and why do it yourself?
It means deliberately sending the prompt an input it was not designed for, such as a missing ticket or a section that does not exist. Like a crash test, it makes the failure happen in the lab instead of in front of an employee.2. Why does a prompt invent an answer for a section that does not exist?
The prompt told it to name the section but not what to do when the section is missing. When a prompt is silent about a case, the model fills the gap with something plausible, because it writes what is likely, not what has been checked.3. What is a test set, and what is a pass count?
A test set is a fixed list of questions, each with what you expect back, that you run each time the prompt changes. The pass count is how many cases got a reply matching what they expect, so versions can be compared fairly.4. What three checks does a case in the test set pass?
The reply must be valid JSON, its `needs_human` must match the case, and its answer must contain the word the case expects, if it expects one. A FAIL line says which check failed.5. Why does the helper have an `ask_helper` function?
It wraps the whole helper in one function, so `run_test_set.py` can call it once per case. A script that only runs from top to bottom cannot be called that way.6. Why change just one thing between test runs?
Two changes at once hide which one helped and which one hurt, like replacing a bicycle's tyres and seat together. One change per run means the difference in the count belongs to that change.7. Why run the test set two or three times?
Models sometimes produce different output from the same input, so a case can pass on one run and fail on the next. In our runs "lost laptop" once failed with `needs_human` true while two other runs passed 9 of 9, so a single run can be lucky.8. What kinds of cases should a test set include?
Typical cases, edge cases and adversarial cases, according to OpenAI's evals guidance. Add the bad inputs you found while breaking the prompt, so the same failure is caught if it comes back.9. What is a prompt library, and how does the helper pick a version?
A folder of named, versioned prompts kept beside the code, with a changelog. The `PROMPT_VERSION` setting, or an environment variable of the same name, picks the file `system-10. How do you compare two prompt versions fairly?
Run the same test set on each, more than once, changing only the prompt file with `PROMPT_VERSION`. Then keep the version with the higher count and write what changed in the changelog.11. Why does a model change also count as a change to test?
OpenAI recommends pinning a production application to a specific model snapshot so behaviour stays consistent, and building tests that let you monitor performance as you change or upgrade model versions. A new model is a new variable, so run the set again.Model answer: how do you find out whether a prompt really works, improve it safely and keep it?
The core question this session answers: “A teammate asks: the Copilot’s prompt gave a great answer once. How do you find out whether it really works, how do you improve it safely, and where do you keep it?”
Four beats, in this order: break it · measure it · change one thing · keep it in a library. Skip any of these four and the answer is incomplete.
- Break it: send bad inputs, such as a missing ticket or a section that does not exist, and see where it fails.
- Measure it: run the test set with the expected result beside each question, and record a pass count.
- Change one thing: edit one rule, re-run, and compare the count; run it more than once, since one run can be lucky.
- Keep it in a library: named, versioned files beside the code, with a changelog.
The full answer, said aloud:
(Break it) “One good answer shows very little, so I would try to make it fail. I would ask about a ticket that does not exist, a policy section that does not exist, and a question outside the help desk topics, and note what each does.
(Measure it) Then I would run the test set, with what I expect written beside each question, and record a pass count.
(Change one thing) I would fix one problem at a time with a rule in the prompt, re-run the set and compare the count. I would run it more than once, because a case can pass one run and fail the next, and I would keep a record so a change that hurt shows up instead of hiding.
(Keep it in a library) And I would save each version of the prompt as a named file beside the code, with a changelog line, so a teammate can review it and we can go back to a version that worked.“
Missing a beat is a failure state. The follow-up question lands where you went quiet: finish without the last beat and the next question is where the prompt lives and how you would restore it. Practise until the four come out without prompting, then stop talking and let the questions come.
Why this matters: where do you use prompt tests and a prompt library at work?
In a sign-off meeting: people who build with models are asked how you know a prompt works. A pass count over several runs gives you a number instead of an opinion, and a library gives you the history behind it.
In a code review: a prompt change shows up as a new file and a changelog line, with a count, the same way a code change shows up with a test. Teams that skip this argue about feelings; teams that keep it argue about numbers.
When a bot that worked in a demo misbehaves in use: a case that passes now and fails later is the signal that a prompt is weak at that spot. A test set you can re-run tells you whether a fix helped or just got lucky.
In the rest of this course: session 9 moves the Copilot onto LangChain and re-runs your test set to check nothing broke; the library’s prompts become prompt templates there; and the evaluation sessions score the test set with a model that grades the answers.
After this unit: what is the assignment and what comes next?
Assignment. Break a prompt with one bad input, then fix it with a rule. The brief for the project work, running the test set on each version and adding a third, is the project file. Keep your test set and your library: the next session starts from them.
What’s next. Session 9 introduces LangChain v1: chat models and prompt templates, composing steps with LCEL, and typed replies, so the helper’s prompt becomes a reusable template.
Want help building this with someone?
Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.
Ask about mentor-led trainingTechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.