Courses / Agentic AI from Scratch (Python)
Session 39 of 40
Capstone build 2: architecture diagram, evaluation report and security assessment
Notes
How to Evidence an LLM App: Evaluation Report, Security Assessment and a Deliverables Check
Agentic AI Foundations · Session 39
Before you start: what do I need before I produce evidence for my capstone?
You need the code folder for this session open in your terminal, and every command below runs from inside it, as the folder’s README.md does. It holds the Copilot, the API and the four data packs from sessions 37 and 38, the grader and attack script carried back from sessions 34 and 35, and four new scripts: code/reports.py, code/score_capstone.py, code/security_check.py and code/check_deliverables.py. Choose the pack you chose in session 38 with export DATA_PACK=banking, using your own pack name, and set LLM_BASE_URL, LLM_API_KEY, LLM_MODEL and EMBED_MODEL as in earlier sessions. Nothing new needs installing. Build the pack’s index and ticket database with python build_index.py and python make_tickets_db.py. Have paper or a notes app ready for the flashcards and the model answer, and a notes/ folder you will create next to the scripts.
Summary: how do I prove my AI assistant works and is protected?
A claim that it works is not evidence. Evidence is something a stranger can open and check without trusting you. In this session scripts produce the facts: code/score_capstone.py asks every question in your pack’s test set and writes reports/evaluation.md, and code/security_check.py runs the attacks and the key scan and writes reports/security.md. code/check_deliverables.py then lists the five capstone deliverables as pass or fail, each with its evidence. The scripts show that evidence exists and has its parts. They cannot show that it is good, so you write the architecture diagram, the evaluation report and the security assessment in your own words, in a notes/ folder.
Key concepts: what do the terms for AI evidence mean?
| Term | Definition |
|---|---|
| Evidence | A fact a stranger can open and check without trusting you, such as a report a script wrote. |
| Test set | The questions of a pack’s test_set.json, each with the answer a person expects. |
| LLM judge | A second model that reads the question, a reference answer and the Copilot’s answer, and scores it from 1 to 5 with a reason. |
| Pass mark (threshold) | The share of answers that must pass, written down before the run. |
| Typical, edge and adversarial cases | Normal questions, unusual ones such as off-topic, and ones that try to get something the caller should not have. |
| Support check | A check that asks whether an answer is backed by the text the search returned. |
| Adversarial test | A test that deliberately sends an attack, run again after each change. |
| Trust boundary | A line on a diagram that shows where different parties are in control. |
| Deliverable | One of the five things the capstone hands over: app, diagram, evaluation report, security assessment and deployment plan. |
Explanations: how do the evaluation, the security check and the checklist work?
What are the five capstone deliverables, and what is the evidence for each?
| # | Deliverable | Evidence |
|---|---|---|
| 1 | A working app | An answer with a source, and a ticket that waited for a person, in audit_trail.jsonl |
| 2 | An architecture diagram | Your notes/architecture.md |
| 3 | An evaluation report | reports/evaluation.md and your notes/evaluation-report.md |
| 4 | A security assessment | reports/security.md and your notes/security-assessment.md |
| 5 | A deployment plan | Made in session 40 |
Think of receipts and a budget. The receipts are the facts, and the budget explains them. The reports in reports/ are receipts: the scripts make them and replace them each time you run them, so a report describes the last run. The files in notes/ are the budget: what the numbers mean, which failures matter and what you changed. A stranger should be able to follow your notes without trusting you.
How does the evaluation run grade my Copilot?
code/score_capstone.py asks every question of your pack’s test_set.json. For each answer it runs four checks.
| Check | What it does | Needs a model |
|---|---|---|
| Judge | grade in code/grader.py scores the answer against the reference, 1 to 5 |
Yes |
| Code check | Looks for the key word, and for a named policy in a policy answer | No |
| Support check | Asks whether a policy answer is backed by the text the search returned | Yes |
| Agent check | Compares the agents and tools used with the expected ones, and checks no ticket was written | No |
A new ticket is refused automatically during the run, so it never waits for you and never changes the ticket database. The role is the pack’s default role, because the cases are written for it. Each question makes several model calls, so the run takes a while.
A judge model can score answers more cheaply and at larger scale than a person. Its known weaknesses include position bias and a taste for longer answers, so its agreement with human labels should be checked before you rely on it. The code check and the agent check are plain code and cannot be talked round. A question that fails the code check and passes the judge is the one to read first. The idea behind the support check is faithfulness: an answer is faithful when its claims can be supported by the retrieved text.
Why must the pass mark be set before the run?
THRESHOLD sits at the top of code/score_capstone.py. It is the share of answers that must pass the judge, and the judge passes a score of 4 or 5, as PASS_SCORE in code/grader.py says. Evaluation guidance recommends a pass or fail threshold in addition to a numerical score. Write it down before the run and change it only before a run, never after you see the result: a mark set after the result is a mark chosen to pass. OpenAI’s evals guidance also notes that a model can give a different output from the same input, so run the evaluation a second time. One run is not proof.
How do I read reports/evaluation.md?
Check four things. First, the pass mark and the result line: was it set before the run, and was it met? Second, the table by kind of question. It separates typical questions from the restricted question and the off-topic one, and a pack that passes only the easy kind has not shown much. OpenAI’s evals guidance says test data should include typical cases, edge cases and adversarial cases, and the kinds in test_set.json cover all three. Third, the failures, each with the judge’s reason. Read the answer behind one yourself with python ask_copilot.py "<the question>" and decide whether the judge is right. Fourth, remember the judge is a model: where the code check disagrees with it, read that answer first. code/reports.py writes the header with the pack and the date, builds each table and saves each report into reports/.
How does the security check work, and what does it leave out?
code/security_check.py runs the attacks of code/injection_attacks.py, each now a function that gives back its result, then the key scan of code/check_secrets.py, and reads your risk register.
| Attack | Defence | Needs a model |
|---|---|---|
| 1 A direct override | The input check | No |
| 2 The override in new words | The prompt rule | Yes, with --live |
| 3 A planted line in a policy file | The scan that drops the chunk | No |
| 4 A read-only caller creates a ticket | The ticket server’s permission lock | No |
| 5 The default role asks for the restricted file | The audience filter | The filter check no; the real search needs --live |
Each outcome is STOPPED, NOT STOPPED or FOR A PERSON TO READ. Attack 2 is for a person, because no script can decide whether a model’s reply is safe, so you read it. The report says which attacks were run and which were not, so a run without --live cannot pass for a full one. It also lists the risks of the register that are still open, and what the check does not show:
- A phrase list catches known wording only; reword an attack and it may pass the first layer.
- Five attacks are a sample. Write down the attacks you tried that worked against you, and what you changed.
- A model can be fooled in ways no script here tries, so a clean report is a floor, not a guarantee.
OWASP says it is unclear whether fool-proof methods of prevention exist for prompt injection, so a security assessment that lists nothing open has not been read hard. Try two attacks of your own against your pack, such as a question that asks for the fraud notes, the clinician notes or the supplier price terms in different words, and write down each one.
What does check_deliverables.py check, and what can it not tell me?
It prints five rows, each PASS or FAIL with an evidence sentence that names the file, or what is missing.
| Row | What the check looks for |
|---|---|
| 1 App | The index and ticket database built; an answer with a source for this pack in audit_trail.jsonl; a ticket that waited for a person |
| 2 Diagram | A mermaid block in notes/architecture.md, at least four arrows, and the word trust |
| 3 Evaluation | The evaluation report for this pack, a threshold and a failure named in your notes, and an adversarial or edge question in the test set |
| 4 Security | The security report for this pack, and the word open in your notes |
| 5 Plan | Owner, restart, secret and cost in notes/deployment-plan.md |
A Mermaid flowchart is made of nodes and edges, and labelled arrows make it readable. A trust boundary on a diagram shows where different parties are in control, so mark what a caller can reach and what only the server can reach:
Caller --key--> API --role from key--> Copilot
Copilot --search by audience--> Documents
Copilot --propose ticket--> Person approves
(a trust boundary between the caller and the API)
In your own file, write this as a Mermaid flowchart inside a mermaid block, with a label on each arrow.
The check shows that the evidence exists and has its parts. It cannot say whether the evidence is good. Expect failures at first, read each evidence line, write your three notes from the reports, and run it again. The last row stays red until session 40. Then ask a classmate to read your three notes as a stranger would and tell you what they could not follow.
What goes wrong when the evidence is careless?
- The pass mark moves after a run. The result looks good because the bar moved. Fix: write
THRESHOLDfirst and change it only before a run. - Attack 5 shows NOT STOPPED. The default role can read the restricted file. Fix: check that
restricted_filesinpack.jsonnames a role, and that the default role’s audiences do not include it. - Row 1 of the checklist fails with the files there. The index was built with a different
DATA_PACKfrom the one you ask with. Fix: run the build steps again with the same setting, from the folder that holds the scripts.
| What you learned | The design decision | Where it shows |
|---|---|---|
| A claim is not evidence | A script writes a report a stranger can open | reports/evaluation.md, reports/security.md |
| A bar set afterwards proves nothing | Fix the pass mark before the run | THRESHOLD in code/score_capstone.py |
| A judge is a model | Back it with plain-code checks, and read failures | code/grader.py |
| A clean run is a floor | Say what is open and what was not run | code/security_check.py |
| A script cannot judge quality | A person reads every piece | code/check_deliverables.py |
Flashcards: test yourself on AI evidence
Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.
1. What does `score_capstone.py` do?
It asks every question of your pack's `test_set.json`, grades each answer with a judge, a code check, a support check and an agent check, and writes `reports/evaluation.md`.2. Why is `THRESHOLD` written before the run?
So the result cannot move it. A pass mark chosen after seeing the result is chosen to pass.3. Why run the evaluation twice?
A model can give a different output from the same input, so the same question can score differently. One run is not proof.4. Why is a judge not enough on its own?
A judge is a model: it can be wrong and has biases such as favouring longer answers. The code check and the agent check are plain code, so a question that fails them and passes the judge is the one to read first.5. Which kinds of question should the test set include?
Typical cases, edge cases such as an off-topic question, and adversarial cases such as a question the default role may not read.6. What are the three outcomes of an attack in `security_check.py`?
STOPPED, NOT STOPPED and FOR A PERSON TO READ. The last is used for the reworded attack, whose reply a person has to judge.7. What does running without `--live` leave out?
The reworded attack and the live half of attack 5, which call the model. The report says they were not run.8. Why can a clean security report not be called safe?
A phrase list catches known wording only, five attacks are a sample, and a model can be fooled in ways no script tries. A clean report is a floor, not a guarantee.9. What does `check_deliverables.py` show, and what can it not show?
It shows whether each deliverable's evidence exists and has its parts. It cannot say whether the evidence is good: a person has to read it.10. What must `notes/architecture.md` contain for row 2 to pass?
A `mermaid` block, at least four arrows, and the word trust, for the trust boundary.Model answer: how do I explain the evidence for an LLM app in an interview?
The core question this session answers: “A reviewer asks: how do you know the Copilot is good and safe, and what are you still unsure about?”
Four beats a reviewer listens for, in this order: a pass mark set first · results by kind · attacks and what got through · what is still open. Skip any of these four and the answer is incomplete.
- A pass mark set first: the threshold written before the run.
- Results by kind: typical, restricted and off-topic questions, with failures.
- Attacks and what got through: the attack results, and my own.
- What is still open: open risks, and what a script cannot show.
The full answer, said aloud:
(A pass mark set first) “I wrote the pass mark down before the run, so the result could not move it. I ran the whole test set twice, because the same question can score differently.
(Results by kind) The report separates typical questions, the question the default role may not read and an off-topic one, and it lists each failure with the judge’s reason. I read the answer behind a failure to see whether the judge was right.
(Attacks and what got through) I ran the five attacks against my pack and tried two of my own. I wrote down which were stopped and which worked.
(What is still open) The report lists the risks not yet controlled. A script cannot decide that a model’s reply is safe, so a person reads the reworded attack. A clean run is a floor, not a guarantee.“
Missing a beat is a failure state. The follow-up lands where you went quiet: skip the last beat and you are asked what you are not sure about. Practise until the four come out without prompting, then stop talking and let the questions come.
Why this matters: why does evidence matter at work?
In a conversation with a teammate or manager: “it works” starts an argument, and a report with a pass mark, failures and open risks ends it. Saying what you did not test is what earns trust.
On the job: launch reviews, audits and handovers ask for something a stranger can open. A script that rebuilds the report after every change keeps that evidence current.
In the rest of this course: session 40 uses reports/evaluation.md in the final demo, adds the deployment plan, and runs the checklist until all five rows pass.
After this unit: what do I do after this lesson, and what comes next?
Assignment. Set DATA_PACK to your pack and build it. Ask one policy question and the pack’s mixed question, answering yes. Run python score_capstone.py and read reports/evaluation.md, run it a second time, then run python security_check.py and python security_check.py --live, read reports/security.md and try two attacks of your own. Run python check_deliverables.py, then write notes/architecture.md, notes/evaluation-report.md and notes/security-assessment.md from your reports and run it again. Project step. Write code/reports.py, code/score_capstone.py, code/security_check.py and code/check_deliverables.py (the build prompts are code/prompts/1-write-reports.md to code/prompts/4-write-check-deliverables.md). The brief for the project work is the project file. Next session: the deployment plan, the final demo and your next step.
Want help building this with someone?
Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.
Ask about mentor-led trainingTechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.