Chapter 8 — Capstone: Build, Demo, Review

Course: GenAI and Agents Fast Track Prerequisites: the three course projects (a PDF question-answering assistant with citations and its evaluation, a research helper with an agent loop, a graph and MCP tools, and a feature added to a repository with a coding agent); the routine of a context file, a plan, small steps with tests and a diff review; the evaluation measures hit rate and faithfulness.

What the reader can do after this unit:

• Write an extension brief that names one goal, at most three files, what is out of scope and the check that proves it
• Measure a baseline before a change, then report before-and-after figures and a passing test count for a build
• Give a demo whose every claim is paired with evidence and whose last part states a limit, then score it against a five-criterion rubric and answer each review finding with a reason

Core question this unit answers: How do you show a GenAI project so that a reviewer can check what you claim?

Connections:

• Builds on: the evaluation measures, the agent loop with its step limit, MCP tools and the coding-agent routine of the earlier projects
• Leads into: showing the finished evidence pack to a hiring reviewer or a team, and applying the same brief, baseline and rubric habits to the next project you build

Student Notes — Chapter 8: Capstone: Build, Demo, Review

GenAI and Agents Fast Track · Chapter 8

Before you start

You need the repository you have built through the course (Python 3, pytest and anyio installed), git and a terminal. Tasks that build an extension are easiest with a coding agent that reads AGENTS.md, as in the previous chapter; any such agent will do. You do not need a model key or a network connection for the build tasks if you keep your tests on the fake model client the repository already uses. A refresher of the three projects follows in section 1, so you can choose without re-reading earlier chapters.

Summary

A capstone is one small extension to a project you already built, finished well enough that someone else can check it. This chapter teaches the four habits that make that possible: write a short brief before you build, measure a baseline before you change anything, pair every claim in your demo with evidence, and score the result against a rubric that a reviewer can also use. You work with the same routine as the previous chapter (context file, plan, small steps with tests, diff review) and add the habits that turn a working program into something you can defend. By the end you can build one extension, show it honestly in a four-part demo, and answer a reviewer’s findings with a reason.

Key concepts

Term	Definition
Capstone	One extension to a course project that you choose, build, demonstrate and have reviewed. It is a small, finished piece of work, not a new large system.
Extension brief	A short written plan for the capstone: one sentence of goal, at most three files you will touch, what is out of scope, the check that proves it works, and the baseline you will measure first.
Out of scope	The list of things you decide not to do in this build, written down so that neither you nor a coding agent adds them later.
Baseline	A measurement taken before you change anything, such as a hit rate, a count of false refusals or the number of passing tests, so that “better” can be shown as a difference between two numbers.
Check	Something you can run that returns a pass or a number: the test suite, an evaluation run or a step count. Without one, “it works” is an opinion.
Claim	One statement your demo makes about the project, such as “the right page is found for 7 of 10 answerable questions”.
Evidence pack	A short file that lists each claim beside the output that backs it (a figure, a test count, a diff) and where the output can be found.
Demo	A short showing of the build in four parts, in order: the problem in one sentence, a run, the numbers, and the limits.
Limit statement	A plain sentence about something the build does not do or does badly, backed by an example from your own run.
Rubric	A list of criteria with a description of what each score means, so two reviewers can score the same work and reach nearly the same total.
Review finding	One observation from a reviewer, quoting the evidence it rests on. You answer each with “accepted” or “rejected” and one reason.

Explanations

1. Choose one extension and write the brief

First, a failure (an invented example). A learner, Asha, starts the capstone with the PDF assistant and plans a better retriever, a web page for it, two new agent tools and a rewrite of the prompts. She writes no brief. Some of it half-works, nothing is measured, and the demo script says “it answers questions about any PDF, correctly, every time”. A reviewer asks one question: “Which PDFs, and how many times did it get a page wrong?” She has no answer, because she did not count. The cause is not a lack of skill. Nothing limited the work, and nothing said what “correct” would mean.

A brief fixes both. Think of the recipe card for a dish you will cook for guests: the dish, the ingredients, the steps and how you will know it is done. Or a builder’s job sheet: which wall, which tools, what is not part of this job. The brief for a capstone has five parts:

1. Goal in one sentence, in the form “make X do Y”.
2. Files you will touch, at most three.
3. Out of scope: at least two things you will not do.
4. Check: the test or the evaluation figure that proves the goal.
5. Baseline: the number you will measure before you change anything.

A brief for the PDF assistant might read:

Goal: make the assistant refuse questions the document cannot answer, instead of guessing.
Files: app/rag/guardrails.py, tests/test_guardrails.py
Out of scope: a new retriever, a web page, new prompts.
Check: false refusals and wrong answers on my evaluation set; the whole suite passes.
Baseline: hit rate, false refusals and passing tests, measured before the first edit.

You choose one of the three projects. The table says where each one starts and what you can measure; the file names are the ones in the repository you have built.

Path	Starting files	A figure you can report
A. The PDF assistant (chapters 3 and 4)	app/rag/evaluate.py, app/rag/eval_set.jsonl, app/rag/guardrails.py	Hit rate, faithfulness, false refusals
B. The research helper (chapters 5 and 6)	app/agent/loop.py, app/agent/team.py, app/mcp_server/server.py	Steps used for a task, tool errors reported, tests passing
C. The repository and its coding agent (chapter 7)	AGENTS.md, app/mcp_server/server.py, the tests folder	Tests passing, files changed outside the plan

The plan rule is the one from the previous chapter: if you could describe the whole diff in one sentence, you may skip the written plan. A capstone is bigger than one such change, so it always gets a brief and a plan.

2. Measure a baseline before you change anything

Think of weighing yourself before you start a training plan. Without the first number, “I got fitter” is a feeling. Or think of photographing a room before you move the furniture: the photo is how you prove what changed. A baseline is that first number for software.

Three measures cover the three paths. Each is a fact from earlier chapters, restated here:

• Hit rate is the number of answerable questions whose expected page was among the retrieved pages, divided by the number of answerable questions.
• Faithfulness of one answer is the supported claims divided by its claims; the faithfulness of a run is the average over the answers the assistant gave. A refusal is not an answer and is left out.
• Tests passing is the count in pytest’s summary line. The count after your work should be the starting count plus the tests you added.

A worked example, with invented figures. A run on 10 answerable questions finds the expected page for 5 of them: the baseline hit rate is 5 / 10 = 0.5. After a change, 7 of 10 hit: 7 / 10 = 0.7. The change added 2 tests, and the suite went from 12 passing to 14 passing. Those three before-and-after pairs are the proof. If the suite had shown 13, one of three things happened: a test was deleted, a test was skipped, or you added fewer than you thought. Find out which before you claim anything.

Here is the function that computes the first two figures, from the evaluation chapter’s code; it is read-only, so use it as it is.

def summarize(results):
    answerable = [r for r in results if r["answerable"]]
    answered = [r for r in results if not r["refused"]]
    hits = sum(hit(r["expected_page"], r["retrieved_pages"]) for r in answerable)
    faith = [faithfulness(r["supported"], r["claims"]) for r in answered]
    return {"hit_rate": hits / len(answerable),
            "faithfulness": sum(faith) / len(faith)}

Take the baseline first, run it, and save the output as a file. Then do the work with the routine from the previous chapter: a context file, a plan you edit down, small steps each with a test, and a read of every diff. Do not edit an existing test to make the suite pass.

3. Turn the build into evidence, then into a demo

First, a failure. Asha’s demo says “it works every time”. The evaluation run behind it covered eight questions, and one of them got the wrong page. A reviewer who sees the run table spots the miss straight away, and from then on stops trusting the other sentences. One unsupported claim spoils the supported ones.

An evidence pack prevents that. Think of the receipts you attach to an expense claim: each amount is paired with the paper that backs it. Or a lab notebook, where a result without its measurement does not count. For the capstone, the pack is one short file with a table: the claim, the evidence and where it lives.

Claim	Evidence	Where
The right page is found for 7 of 10 answerable questions	Hit rate 0.7 after, 0.5 before	results-after.txt, results-before.txt
No existing test was changed	The diff touches only the planned files	change.diff
The suite passes with two more tests	14 passing, up from 12	suite-output.txt

Words like “always”, “never” and “every” are claims about every case, and a run of ten questions cannot back them. Write the figure instead: “7 of 10”. If a sentence has no evidence, cut it or state the weaker version the figures do support.

Now the demo. Think of a cooking show: the dish is named, it is cooked in front of you, it is tasted, and a good cook says what they would change. Or think of a test drive: you drive the car, read the figures and hear what the car cannot do. A capstone demo has four parts, in this order:

1. The problem in one sentence: what was wrong or missing.
2. A run: the feature working on one real example, live or recorded.
3. The numbers: the before-and-after figures from your evidence pack.
4. The limits: at least one thing it does not do, with an example from your own run.

The last part matters most to a reviewer and is the one beginners skip. Think of a job application that says “expert” for a skill nobody tested: the claim costs you trust. A plain limit (“it still refuses one answerable question, q4, because the best chunk scored just under the threshold”) makes the other three parts believable.

4. Review against a rubric

Think of a driving test: the examiner holds a checklist, so two examiners reach almost the same result. A rubric is that checklist. Or think of an exam mark scheme, where each mark is tied to something you can point at. The capstone rubric has five criteria, each scored 0, 1 or 2, for a total from 0 to 10.

Criterion	Score 2	Score 1	Score 0
Scope	A brief with at most three files, an out-of-scope list and a check	A brief missing the out-of-scope list or the check	No brief, or the work grew past it
Proof	A baseline and an after figure from your own runs, and the suite passing with the starting count plus your tests	Tests pass but there is no before figure, or the counts do not add up	Nothing was run
Review	The diff read against the plan, no existing test weakened, five review answers written	The diff skimmed, or one review answer missing	Only the agent’s summary read
Honesty	Every claim matches a figure in the pack, and a limit is stated	One claim goes beyond the evidence, or no limit is stated	Several claims have no evidence
Clarity	Four demo parts in order, with a one-sentence problem	A part missing or out of order	The demo cannot be followed

The total decides what you do next: 8 to 10 is ready to show; 5 to 7 means fix the lowest-scoring criteria and score again; 0 to 4 means rebuild the weakest part first. Worked example: scores of 2, 2, 1, 0, 1 total 6. The lowest criterion is Honesty at 0, so fix the claims first, then re-score.

Writing and answering a finding. A good finding quotes its evidence and names what it breaks: “The pack claims no refusals on answerable questions; row q4 shows a refusal. This breaks the Honesty criterion.” You answer each finding with accepted or rejected and one reason. Accept a finding that affects correctness or something your brief says you would deliver. Reject one that is a preference or outside your scope (“add caching”, when the brief lists speed as out of scope), and note it as future work. Rejecting is not ignoring: the reason is part of the review.

A second opinion from a fresh agent session. Official guidance for one coding agent describes a writer and reviewer pattern: a fresh context improves review, because the reviewer has no bias toward code it just wrote. The same guidance warns that a reviewer asked to hunt for gaps will tend to report something even when the work is sound, so tell it to flag only gaps that affect correctness or your stated requirements. Give it your brief, your diff and the rubric, and treat what it says as findings to accept or reject. It adds to your own review and does not replace it.

5. Three situations, then the order of work

• A learner finishes the build, the suite is green, and the demo claims “no wrong answers”. The evaluation file shows two. The claim has no evidence, so it is cut or rewritten as the figure the run supports.
• A reviewer says “add a reranker”. The brief lists retrieval changes as out of scope, and the finding does not point at a failure. Reject it with that reason and record it as future work.
• A learner starts the change without measuring and later cannot say whether the hit rate moved. Without a baseline, “better” cannot be shown; restore the earlier version in a scratch copy, measure it, and compare.

If this happens	Do this	Why
You are about to start building	Write the brief and take the baseline	Scope and a “before” number cannot be added afterwards
The suite count is lower than start plus added	Find the missing or skipped test	A test may have been deleted or weakened
A sentence in the demo has no evidence	Cut it or write the weaker, supported version	One unsupported claim damages the rest
A finding is a preference	Reject it with a reason	The brief sets the scope

The order of the capstone is: brief, baseline, context file and plan, small steps with tests, diff review, evidence pack, demo, rubric score. The next section lets you practise it by speaking.

Flashcards

Say the answer aloud before revealing it. Speaking is a different skill from reading; practise both.

1. What is an extension brief?

   • A short plan written before the build: one goal sentence, at most three files, what is out of scope, the check that proves the goal and the baseline you will measure first. It limits the work and says what “done” means.

2. Why write what is out of scope?

   • A list of things you will not do stops the work, and any coding agent you use, from adding a refactor or a feature nobody asked for. It also gives you a reason to reject a reviewer’s finding that is only a preference.

3. What is a baseline, and when do you take it?

   • A measurement such as hit rate or passing tests, taken before you change anything. Without it you cannot show a difference, and “better” stays an opinion.

4. How do you calculate hit rate?

   • Count the answerable questions whose expected page was among the retrieved pages, and divide by the number of answerable questions. Questions the document cannot answer are not in the count.

5. What should the test count be after you add three tests to a suite that passed 20?

   • Twenty-three. If the suite shows fewer, a test may have been deleted, skipped or weakened, and you find out which before you claim anything.

6. What is an evidence pack?

   • A short file with a table that pairs each claim in your demo with the output that backs it and the place the output is saved. It lets a reviewer check a claim without asking you.

7. What do you do with a claim that has no evidence?

   • Cut it, or rewrite it as the weaker statement your figures support, for example “7 of 10” instead of “every time”. One unsupported claim makes a reviewer doubt the supported ones.

8. What are the four parts of a demo, in order?

   • The problem in one sentence, a run on one real example, the numbers from your evidence pack, and the limits. The limits come last and make the first three believable.

9. How does the rubric turn a score into a next step?

   • Five criteria are scored 0 to 2. A total of 8 to 10 is ready to show, 5 to 7 means fix the lowest-scoring criteria and score again, and 0 to 4 means rebuild the weakest part first.

10. How do you answer a review finding?

    • With “accepted” or “rejected” and one reason. Accept it if it affects correctness or something your brief promised; reject it with a reason if it is a preference or out of scope, and note it as future work.

11. Why use a fresh agent session as a reviewer, and what must you tell it?

    • A fresh context is not biased toward code it just wrote. Tell it to flag only gaps that affect correctness or your stated requirements, because a reviewer asked for gaps will usually report some even when the work is sound.

Model answer

The core question this unit answers: “How do you show a GenAI project so that a reviewer can check what you claim?”

Five beats a good answer covers; hit all five, in this order:

1. Scope — one extension, written as a brief with files, out-of-scope items and a check
2. Baseline — a measurement before the first change
3. Evidence — every claim paired with a figure or output from your own runs
4. Limits — at least one stated limit, with an example
5. Review — a rubric score, and each finding answered with accepted or rejected and a reason

The full answer, said aloud:

(Scope) “I pick one extension and write a brief first: one goal, no more than three files, what is out of scope and the check that proves it, so the work stays small enough to finish.

(Baseline) Before I change anything I measure, for example the hit rate and the number of passing tests, so that better is a difference between two numbers.

(Evidence) Every claim in my demo sits beside the output that backs it. If I have no figure for a sentence, I cut it or write the weaker version the figures support.

(Limits) I state what it does not do, with an example from my own run, because that is what makes the rest believable.

(Review) I score it against a rubric, and I answer every finding with accepted or rejected and a reason, so a reviewer can see how I judged the feedback.“

Skip any of these five and the answer is incomplete. A reviewer or interviewer who hears only “I built it and it works” will ask what you measured before, what the numbers are and what it cannot do. Practise until the five beats come out in order without notes.

Why this matters

On the job: a teammate who reads your pull request wants the brief, the before-and-after numbers and the limits, because those are what they can check. A claim they can check is a claim they can approve.

In interviews and reviews: “tell me about a project you built” is a standard question. The five-beat answer turns “I built a chatbot” into a project with a scope, a measurement and an honest limit, and it applies to any project, not only these three.

As a builder: a brief and a baseline are cheap to write at the start and hard to recreate at the end. Keeping an evidence pack for each project also leaves you something concrete to show.

After this unit

Hands-on project. Four tasks: (1) audit a sample demo against its evaluation run, computing the figures and labelling each claim; (2) write your brief, take your baseline, build one extension with a coding agent and assemble your evidence pack; (3) write your demo script and score your build against the rubric; (4) optionally, ask a fresh agent session to review your build and answer its findings. The full brief is in the project file.

What’s next. This is the last chapter of the course. Keep your evidence pack, your brief and your rubric score: they are the piece you can show to a hiring reviewer or a team, and the same brief, baseline and rubric apply to the next project you build.

Hands-on Project — Chapter 8: Build One Extension, Show It, Score It

Objective

Choose one of the course projects, extend it in one small step with a measured before and after, show it in a four-part demo whose claims all have evidence, and score it against the rubric in the notes, after first practising the audit on someone else’s demo.

What you need: Task 1 needs Python 3 and a text editor, or only paper and a calculator: no account, no key and no cost. Task 2 needs the repository you built in the course (Python 3, pytest and anyio installed), git and a coding agent that reads AGENTS.md; the agent may charge for its use, so check its plan first. Your tests keep using the fake model client, so they need no model key and no network. Task 3 needs only the files from Task 2. Task 4 is optional and needs a coding agent again. The figures in Task 1 are invented for the exercise and come from no real system.

You write the brief, the claims and the review answers yourself. This brief gives the facts and the checks; the finished files and the right numbers are not in it. Work through the tasks in order and check each output before moving on.

Facts used by the tasks.

• Hit rate is the number of answerable questions whose expected page is among the retrieved pages, divided by the number of answerable questions.
• Faithfulness of one answer is its supported claims divided by its claims. The faithfulness of a run is the average over the answers the assistant gave. A refusal is not an answer and is left out.
• A false refusal is a refusal of a question the document can answer.
• The suite is run with python -m pytest -q, and its summary line gives the number of tests passing.
• The path table in the notes lists the starting files for each of the three projects. In the repository, the evaluation code is app/rag/evaluate.py, the refusal threshold is the min_score argument of guarded_answer in app/rag/guardrails.py (default 0.40), the agent loop is run_agent in app/agent/loop.py (its max_steps argument defaults to 6, and it returns the step count and why it stopped), and the notes server is app/mcp_server/server.py.

---

Task 1 — Audit a sample demo

A learner demonstrated a policy assistant. The evaluation run below covers ten questions on a policy document. “Retrieved pages” are the pages the assistant found; “Refused” says whether it declined to answer; “Claims” and “Supported” count the claims in its answer and how many the retrieved pages support.

Id	Answerable	Expected page	Retrieved pages	Refused	Claims	Supported
q1	yes	1	1, 3	no	3	3
q2	yes	2	2, 5, 6	no	4	4
q3	yes	4	3, 5	no	2	1
q4	yes	5	5, 2	no	3	3
q5	yes	6	6, 7	no	5	4
q6	yes	7	4, 8	yes	0	0
q7	yes	8	8, 9	no	4	4
q8	yes	9	9, 1	no	2	2
q9	no	none	3, 6	yes	0	0
q10	no	none	1, 2	no	2	0

The learner’s demo made these six claims:

• C1. “The assistant finds the right page for every question it can answer.”
• C2. “The hit rate on this run is 0.80.”
• C3. “Faithfulness on this run is above 0.70.”
• C4. “The assistant has never answered a question the policy cannot answer.”
• C5. “The assistant answers quickly.”
• C6. “The assistant refused at least one question the policy cannot answer.”

Compute the hit rate, the faithfulness of the run, the number of false refusals and the number of unanswerable questions that received an answer. Then label each of the six claims supported (the data show it as stated) or not supported (a row in the data breaks it, or the data say nothing about it). For each claim that is not supported, write a replacement claim that your figures do support, or write “cut” if the data cannot support any version.

Scope: ten rows and six claims, nothing more. Stop once the checks hold.

Checks:

• audit.md gives the four figures, with the working shown for each: the counts you divided, not only the result.
• Each of the six claims has a label and the row or figure that decides it.
• Each not-supported claim has a replacement that quotes one of your figures, or “cut” with a reason.
• Faithfulness is rounded to two decimal places and says which answers it averaged over.

Output: audit.md.

---

Task 2 — Write the brief, take the baseline, build one extension

Choose one path from the table in the notes (A, B or C) and write brief.md with the five parts the notes list: goal, files (at most three), out of scope (at least two items), check and baseline. Take the baseline before any edit and save it as baseline.txt: the suite’s passing count, plus at least one figure your path can report (hit rate, faithfulness or false refusals for A; steps used or tool errors for B; files changed outside the plan for C). Then build the extension with the routine from the previous chapter: AGENTS.md in place, a plan you read and edit saved as plan.md, small steps each with a test, and a read of every diff. Run the same figure again afterwards and save it as after.txt. Finally, write evidence.md: a table with one row for each claim you intend to make in your demo, giving the claim, the evidence and the file it lives in.

Scope: one extension that you could describe in one sentence, at most three files changed, no existing test edited. Stop once the checks hold.

Checks:

• brief.md has all five parts, names at most 3 files and lists at least 2 out-of-scope items.
• baseline.txt was taken before the first edit and holds the passing-test count and at least one more figure.
• The only files changed are those the brief names plus new test files, and no existing test file was edited.
• The whole suite passes, and the number passing equals the baseline count plus the tests you added.
• after.txt repeats each baseline figure, so that every figure has a before and an after.
• evidence.md has between 3 and 5 claims, and every claim names a file you can open that backs it.

Output: brief.md, baseline.txt, plan.md, after.txt, evidence.md and your diff saved as capstone.diff.

---

Task 3 — Write the demo and score your build

Write demo.md, a script for a four-part demo of your build: the problem in one sentence, the run you would show, the numbers, and the limits. Use only claims that are rows in your evidence.md, and number them. Then write review.md: score your build 0, 1 or 2 on each of the five rubric criteria in the notes, give a reason for each score that names a file, add up the total and state which band it falls in. If the total is below 8, name the lowest-scoring criterion, make one change that raises it, and write the new score beside the old one.

Scope: one script of at most one page and one score table. Stop once the checks hold.

Checks:

• demo.md has the four parts in the order the notes give, and the problem is one sentence.
• Every claim in demo.md is numbered and matches a row in evidence.md.
• The limits part names at least one thing your build does not do, with an example from your own run.
• review.md has five scores, each with a reason that names a file, and the total is the sum of the five.
• The band stated matches the total, and a total below 8 comes with a named change and a re-score.

Output: demo.md and review.md.

---

Task 4 — A second opinion on your build (optional)

Open a fresh coding agent session, one that did not write your code. Give it brief.md, capstone.diff and the rubric, and ask it to report only gaps that affect correctness or the requirements in your brief. Record the exact prompt. For each finding, write “accepted” or “rejected” with one reason, and change review.md if you accept anything.

Scope: one reviewer session and one answer for each finding. Stop once the checks hold.

Checks:

• review2.md holds the prompt you used and every finding the reviewer reported.
• Each finding is marked “accepted” or “rejected” with one reason that refers to your brief or the rubric.
• review2.md ends with a sentence saying whether the reviewer found anything your own review missed.

Output: review2.md.

Keep brief.md, baseline.txt, after.txt, evidence.md, demo.md and review.md together: this is the last chapter of the course, and that evidence pack is the piece to show a hiring reviewer or a team, and the starting point for the brief of your next project.