Courses / Agentic AI from Scratch (Python)
Session 28 of 40
Lab and Milestone 2: a routed agent with human approval
Notes
How Do You Test a Routed LangGraph Agent Against Milestone 2?
Agentic AI Foundations · Session 28 — Lab and Milestone 2: a routed agent with human approval
Before you start: what do I need before I run the Milestone 2 checks?
You need the code folder for this session open in your terminal. It holds code/check_milestone2.py, code/copilot_graph.py (session 27’s graph, kept as a module), code/data/test_set.json, the prompts in code/model_prompts/ and one build prompt in code/prompts/. Install the libraries with pip install langchain langchain-openai langchain-chroma langchain-community langchain-text-splitters langgraph langgraph-checkpoint-sqlite, set LLM_BASE_URL, LLM_API_KEY, LLM_MODEL and EMBED_MODEL as in earlier sessions, then build the policy index and the ticket database with python code/build_index.py and python code/make_tickets_db.py. Recap of what this session assumes: the Copilot sorts each question into a policy, ticket or other path (sessions 23 and 24), saves each conversation in a SQLite file under a thread id (session 25), pauses for a person’s approval before it writes a ticket (session 26), and checks a policy answer against the policy text before sending it (session 27). Have paper or a notes app ready for the Flashcards and the Model answer.
Summary: what is Milestone 2 and how do you check it?
Milestone 2 is the point where the Copilot is a routed agent with a person’s approval: it is routed, saved, approved and checked. One script, code/check_milestone2.py, proves it by running the checks it defines (six of them) and then a small test set, printing PASS or FAIL for each. Each check passes only when the script measured the behaviour, such as a count of ticket rows, not from a fixed value. When a line fails, a short table points from the symptom to the likely cause and the part of the graph to look at. Treat the test set’s pass count as a guide, read each failure, and run a failing question again before deciding it is a real problem.
Key concepts: Milestone 2 and testing terms explained in plain English
| Term | Definition |
|---|---|
| Milestone 2 | The point where the Copilot is routed, saved, approved and checked, shown by six checks passing. |
| Check | One measured test of a behaviour; it prints PASS only when the script saw the behaviour happen. |
| Test set | A fixed list of questions, each with the path it should take and words its answer should hold. |
| Pass count | How many checks or test questions passed; a guide, not a proof. |
build_graph |
The function in code/copilot_graph.py that compiles a fresh graph on a checkpoint file you give it. |
| Restart check | A check that builds a second graph on the same checkpoint file and reads the first graph’s saved thread. |
| Approval pause | The point where the graph stops before a ticket exists and waits for a person’s yes or no. |
| Unsupported answer | A policy answer that states something the retrieved policy text does not say. |
| Failure table | A table from a symptom to its likely cause and the place in the code to look. |
| Throwaway checkpoint file | A temporary file the script uses so your lesson files are left alone. |
Explanations: how do the Milestone 2 checks work?
What is Milestone 2, and why run checks instead of a demo?
Failure first. A teammate shows the Copilot answering one leave question correctly and says it is ready. A week later a question about a ticket goes down the policy path, and a restart loses a conversation. The demo exercised one path; nothing exercised the rest.
Think of a driving test. The examiner does not ask whether you feel ready; they run a list of manoeuvres and mark each. A pre-flight checklist works the same way: every item is measured, not assumed. Milestone 2 is that list for the Copilot, and it makes four promises: routed (each kind of question takes its own path), saved (a conversation survives a restart), approved (a ticket waits for a person’s yes) and checked (an unsupported policy answer is not sent). The picture of the whole graph is the work of sessions 23 to 27: receive, remember_fact and sort lead to a policy path with check_node, a ticket path with a draft and an approval, and a fallback, all ending in save_turn, with a checkpointer and a store behind them.
What are the six checks?
code/check_milestone2.py defines six checks and runs them in order:
| Check | What the script measures | Part of the graph |
|---|---|---|
| 1 Policy path | A leave question takes the policy path and the answer names a source | sort, policy_node |
| 2 Ticket path | A question about T-1042 takes the ticket path | draft_node, lookup_node |
| 3 Fallback | A weather question gets the fallback | fallback_node |
| 4 Restart | A second graph continues the first graph’s thread | the checkpointer |
| 5 Approval | A ticket waits for a person; no writes nothing, yes writes one | approval_node |
| 6 Unsupported answer | An answer the policies do not support is not sent | check_node |
Each check measures something and prints PASS only when it saw it. Checks 1 to 3 read the kind the graph chose and the words in the answer. The script uses a throwaway checkpoint file in a temporary folder, so nothing is left in your lesson files except one new row in the tickets table.
How does the script restart the Copilot?
Picture closing a notebook and opening it tomorrow: what you wrote is still on the page. A conversation saved in a SQLite file works that way, while one held in memory is gone with the program. To test this, code/copilot_graph.py wraps the compile step in a function:
def build_graph(checkpoint_file=CHECKPOINT_FILE):
saver = SqliteSaver(
sqlite3.connect(str(checkpoint_file),
check_same_thread=False))
return builder.compile(checkpointer=saver, store=store)
Check 4 calls it a second time on the same file, which acts like a program that was stopped and started again, then reads the thread the first graph saved:
restarted = build_graph(checkpoint_file)
saved = restarted.get_state(config).values
It asks a follow-up on that thread and expects the saved history to grow from one turn to two. The checkpointer uses the thread id to find a thread’s checkpoints, so the same id on a new graph finds the old conversation. The LangGraph docs note that the in-memory saver loses all its checkpoints when the process restarts, which is why this check would fail on it.
How does check 5 measure the approval pause?
A shop assistant who needs a manager’s signature does not start the refund first. The graph does the same: it must stop before the ticket exists. Check 5 counts the rows in the tickets table at three moments:
before = count_tickets()
paused = ask(graph, "m2-reject", "Please raise a ticket: ...")
waiting = "__interrupt__" in paused and count_tickets() == before
graph.invoke(Command(resume="reject"), config)
nothing_written = count_tickets() == before
The graph must be paused with no new row, a reject must still leave no new row, and then an approve on another thread must add one row. PASS needs all three. This is also why the write belongs in a step after the approval: the LangGraph docs say that when a graph resumes, the node that called interrupt starts again from the beginning, so a write placed in that node can run twice and make two tickets.
How does check 6 test the self-check, and what is the test set?
Check 6 uses the demo switch from session 27. With bad_attempts=2, both answers the policy node produces are made up, so the self-check should fail both times and the graph should give up. The check passes when the answer says it cannot confirm and does not hold the made-up number.
After the six checks the script runs the test set, the list of questions in code/data/test_set.json. Each has the kind it should take and a must_contain phrase its answer should hold. A question passes when both match. OpenAI’s evals guidance says test data should include typical cases, edge cases and adversarial cases, and the file mixes policy, ticket and other questions in that spirit.
Like a spelling quiz, a score shows where you stand and a missed word shows what to study. One taste of a recipe can mislead; several tastes show the pattern. The same guidance says models sometimes produce different output from the same input, so a question can fail once because the wording or the label varied. Run the script again: a question that fails every time is a real problem. Milestone 2 is met when all six checks pass; the test set is a guide.
How do you read a failure and fix it?
A doctor facing a cough looks at the likely cause first and tests it. A mechanic reads a warning light as a pointer to where to look, not what to replace. Use the same habit:
| What you see | Likely cause | Where to look |
|---|---|---|
| A ticket question took the policy path | The sort label was wrong | code/model_prompts/sort.md, route_by_kind |
| Check 4 fails | The state did not survive | the checkpointer in build_graph |
| Two tickets for one approval | The write is in the node that asks | approval_node |
| Check 6 fails | The check was skipped or too lenient | check_node, code/model_prompts/check.md |
Change one thing, then run the whole script again. A fix that helps one check can break another, and with two changes at once you cannot tell which one helped.
What goes wrong without these checks, and what does it mean for the design?
Before the design table, three situations you will meet:
- A ticket question is answered from the policies. The sort label was wrong, so the route sent it down the policy path.
- A restart makes the Copilot forget a conversation. The state was held in memory instead of the SQLite file.
- One approval makes two tickets. The write sits in the node that asks, and that node runs again when the graph resumes.
What these mean for the design:
| What you learned | The design decision | Where it shows |
|---|---|---|
| A demo shows one path | Run a check for each path | checks 1 to 3 |
| State must survive a restart | Save to a file and reuse the thread id | check 4 |
| A write needs a yes | Put the write in a step after the approval | check 5 |
| A model’s answer can be unsupported | Check it, then stop after a retry | check 6 |
Basic assignment: run python code/build_index.py, python code/make_tickets_db.py and then python code/check_milestone2.py. The script makes many chat calls, so wait for its final lines before you read the result. For each FAIL, find its row in the failure table, look at the part it names and change one thing before you run again.
Project step: change one thing and see. Add three of your own questions to code/data/test_set.json, one for each path. Break the sort prompt on purpose by deleting the ticket line, and note which checks fail. Run the script twice: do the same questions pass each time, and which ones are close? Run python code/make_tickets_db.py between runs to reset the ticket table; check 5 counts from its own starting point, so the reset only tidies the table. The build prompt is code/prompts/1-write-check-milestone2.md. Stop when all six checks pass on your Copilot.
Flashcards: test yourself on Milestone 2 and testing the Copilot
Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.
1. What is Milestone 2?
The point where the Copilot is a routed agent with a person's approval: routed, saved, approved and checked. The checks in `code/check_milestone2.py` prove it.2. Why run checks instead of trusting a demo?
A demo exercises one path. The checks cover each path, a restart, the approval pause and an unsupported answer, so a break in any of them shows up before release.3. Name the six checks.
Policy path with a source, ticket path, fallback, restart, approval pause, and an unsupported answer not sent. Each prints PASS only when the script measured it.4. Why does `build_graph` exist?
It compiles a fresh graph on a checkpoint file you give it. The script builds a second graph on the same file to act like a restart and reads the thread the first graph saved.5. How does check 5 know a ticket waited for a person?
It counts rows in the tickets table while the graph is paused, after a reject and after an approve. The count must not change until the approval, then rise by one.6. Why must the ticket write come after the approval step?
When a graph resumes, the node that asked starts again from its beginning. A write in that node can run twice and make two tickets, so the write goes in a step after it.7. How does check 6 make both answers wrong?
It passes `bad_attempts=2`, the demo switch from session 27, so both tries are made up. The check passes when the reply says it cannot confirm and lacks the made-up number.8. What does a test question need to pass?
The graph must choose the expected `kind`, and the answer must hold the question's `must_contain` words. Both conditions are tested in the script.9. Why is a pass count only a guide?
A model's label or wording can vary between runs, so a question may fail once and pass next time. Read each FAIL and run it again; a question that fails every time is a real problem.10. Why change one thing at a time when fixing a failure?
A fix that helps one check can break another, and with two changes you cannot tell which one helped. Change one thing and run all six checks again.Model answer: how do I explain Milestone 2 to a teammate?
The core question this session answers: “A teammate asks: how do you know the Copilot is ready, and what do you do when a check fails?”
Four beats a teammate listens for, in this order: what Milestone 2 is · the six checks · reading a failure · fix and re-run. Skip any of these four and the answer is incomplete.
- What Milestone 2 is: routed, saved, approved, checked.
- The six checks: each measures something and prints PASS only when it saw it.
- Reading a failure: find the likely cause and the part of the graph to look at.
- Fix and re-run: change one thing, then run everything again; treat the test set as a guide.
The full answer, said aloud:
(What Milestone 2 is) “It means the Copilot routes each question to its own path, saves a conversation across a restart, asks a person before it writes a ticket, and checks a policy answer before sending it.
(The six checks) One script runs six checks, one for each path and each promise, and each prints PASS only from something it measured, such as a count of ticket rows.
(Reading a failure) A FAIL points to a likely cause and a place in the graph: the sort prompt, the checkpointer, the approval node or the check node.
(Fix and re-run) I change one thing and run all the checks again, and I read the test set’s pass count as a guide, running a failing question twice before I call it a real problem.“
Missing a beat is a failure state. Leave out the checks and the next question is what exactly you ran; leave out the last beat and the next question is how you know your fix did not break something else. Practise until the four come out without prompting, then stop talking and let the questions come.
Why this matters: why do milestone checks matter at work?
In a conversation with a teammate or manager: people ask whether an assistant is ready for a pilot. “Six checks pass, and here is the one that waits for a person’s approval” is evidence, where “the demo worked” is a feeling.
On the job: engineers re-run a test set after each change, auditors ask to see that an assistant asks before it changes anything, and the person on call needs to know which part to open first when a check goes red.
In the rest of this course: sessions 29 to 32 add MCP tool servers and several agents on top of this Copilot, and later sessions turn this kind of check into a larger test and safety practice.
After this unit: what do I do after this lesson, and what comes next?
Assignment. Run code/check_milestone2.py on your Copilot and fix one failing check at a time until every check passes. The brief for the project work, Milestone 2, is the project file. Keep your Copilot files: the next module builds on them.
What’s next. Session 29 starts MCP basics: host, client and server, with tools, resources and prompts that any assistant can use.
Want help building this with someone?
Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.
Ask about mentor-led trainingTechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.