Chapter 11 — Capstone: Ship a Grounded, Tool-Using Feature

Course: Building AI Agents

Prerequisites: You can read and run a short Python script that uses lists, dictionaries, loops, sets and a regular expression. The capstone draws on earlier work: retrieval and citations (Build and Evaluate RAG Applications), tool gates (Chapters 1 and 9), and tracing and cost, host-system limits and the release gate over repeated runs (Taking AI to Production). The notes restate each technique in one line, naming where it was taught, before they use it. Every other term is defined in the notes before it is used.

What the reader can do after this chapter:

• Order the build of a retrieval-grounded, tool-using feature so that a thin end-to-end path exists before any part is hardened.
• Apply a refusal threshold and a citation check to drafted answers, and a written rule set to proposed tool calls, and say what each one decides and what it leaves open.
• Decide whether a release candidate ships by a written rule that combines repeated-run results, a cost cap and trace coverage.

Core question this chapter answers: You must put a feature that answers from company documents and can look up an order into a system that already works: what do you build first, what must be true before you ship it, and what evidence shows that it is?

Connections:

• Builds on: the earlier chapters of this course and the two courses named above, in particular the citation check, the tool allow-list, the trace and cost record, and the release gate.
• Leads into: nothing further. This is the last chapter of the course. This chapter’s evidence file is the last thing you produce; it is what you would hand to someone who was not there when you built the feature.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines. Every accepted value, or the rule that decides it, follows from what the task prints.

Building AI Agents — Chapter 11: Capstone: Ship a Grounded, Tool-Using Feature

Before you start

Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library. You do not need an account, an API key or a network connection: the model’s drafts, its proposed tool calls and the release results are all printed in the project file.

Keep one question in mind while you read: if this feature went live tonight, which of its three promises (it answers from the documents, it touches only what it should, it costs and behaves as agreed) could I prove from a record, and which would I only be hoping?

Summary

A capstone is the point where earlier skills have to work together rather than one at a time, and the order in which you assemble them matters. This chapter builds a feature that answers from company documents and can look up an order, starting with a thin end-to-end path and then hardening each part in turn. The grounding part retrieves passages, refuses when the best match is weak, and checks that each citation names a real chunk that contains the quoted wording. The tool part treats every call the model asks for as a request that your code checks against written rules before anything runs. The release part decides by a written rule over repeated test results, a cost cap and trace coverage, so that “ready” is a calculation and not a feeling.

Key concepts

Term	Definition
Capstone	A final piece of work that combines skills from earlier chapters into one deliverable that is judged as a whole.
Walking skeleton	A tiny implementation of the system that performs a small end-to-end function and links together the main architectural components.
Host system	The existing application that the feature has to live inside, whose owners set the rules it works under (Taking AI to Production).
Chunk	One piece of a document, cut out so that it can be indexed, retrieved and cited on its own (Build and Evaluate RAG Applications).
Retrieval score	For a question and a chunk, the number of different non-stop words they share; the chunk with the highest score is the top chunk (Build and Evaluate RAG Applications).
Refusal threshold	A cut-off on the top chunk’s score: below it, the system says the documents do not contain the answer instead of answering (Build and Evaluate RAG Applications).
Citation check	A code check that a cited chunk id exists and that the chunk’s text contains the quote the answer relies on (Build and Evaluate RAG Applications).
Dangling citation	A citation that names a chunk id that does not exist.
Unsupported claim	A claim whose cited chunk exists but does not contain the quoted wording.
Citation recall and precision	Recall is the share of an answer’s statements that are fully supported by their citations; precision is the share of its citations that support the statement they are attached to (Build and Evaluate RAG Applications).
Tool call	The structured request a model produces when it wants a tool used: a tool name and arguments. Your code decides whether to run it (Chapter 1).
Allow-list	A list of the only tools that may run; a call to anything not on it is denied (Chapter 9).
Argument validation	Checking that a call’s arguments have exactly the expected names and that each value has the expected shape, before the tool runs (Chapter 1).
Call budget	The most tool calls one request may make; later calls in the same request are denied.
Human approval	A person confirming a consequential action before it runs, instead of the model’s call being enough.
Exposed columns	The columns of a host record that the feature is permitted to read; the others are not passed on. A column-level grant is the database’s way of enforcing this (Taking AI to Production).
Trace and span	A trace is the record of one request end to end; a span is one timed operation inside it, which can carry token counts (Taking AI to Production).
Cost in micro-dollars	For a span, input tokens times the input price plus output tokens times the output price, where a price is in dollars per million tokens, which equals micro-dollars per token (Taking AI to Production).
Regression set and release gate	A regression set is a fixed set of cases kept to test every later release; a release gate is a written rule over repeated runs of it that decides whether a release may ship (Build and Evaluate RAG Applications and Taking AI to Production).

Explanations

The failure, before the explanation

A team builds a support assistant that answers refund questions from policy documents and can cancel an order for the customer. The demonstration uses three friendly questions and goes well. Here is what production shows later:

Reply 1   "Refunds go back within 5 days (policy P-9)."      P-9 is not an id in the store.
Reply 2   The model asks for the tool "close_account".        The dispatcher looks the name up, finds nothing,
                                                              and falls back to the nearest tool name.
Reply 3   The model asks to cancel order 1001 on a question   Nobody approved it; the cancel ran.
          about delivery times.
Release   The new prompt passes the three demo questions.     Nobody counted cost, repeated runs or missing traces.

Nothing here is exotic. Each failure is a promise the demo did not test: that a citation points at something real, that a tool call is checked before it runs, and that a release is decided by a rule. The three sections below add one control per promise, after a section on the order in which to build.

Build order: a thin path first, then harden

Take the support assistant. Step 1 is a route in the host system that returns a fixed sentence, writes one trace span and has one test case that passes. Step 2 replaces the fixed sentence with retrieval over two chunks. Step 3 adds the citation check and the refusal. Step 4 adds one read-only tool. Step 5 adds the budget, the approval rule and the cost cap. After every step the test case still passes, so a failure points at the step that was just added.

A name for the first step comes from Alistair Cockburn, whose definition is quoted at Henrico Dolfing’s site: a walking skeleton is “a tiny implementation of the system that performs a small end-to-end function”, and it “should link together the main architectural components” (Henrico Dolfing, Start Your Project With a Walking Skeleton, retrieved 2026-10-04). A second picture: when you plumb a house you run one pipe from the mains to a single tap and see water come out before you fit the bathroom, because a leak in one joint is easy to find when only one joint exists.

• Applied use. Make the route, the trace and one test exist first, then replace the stubs one at a time.
• Real-world signal. A reviewer who was not there can run a plain path that works, and cannot run a polished retrieval stage that has no route to reach it.
• Practical consequence. Because the host system’s route, credentials and limits (Taking AI to Production) are part of the skeleton, you meet them in the first step, when changing course is cheap.
• Versus the alternatives. Building the best retrieval you can first is the distractor: it can be excellent and still not fit the host’s data access. Building everything in parallel is the other, because with five parts changing at once a failure cannot be placed.

Grounded answers: retrieve, check, refuse

Take a facilities assistant with one chunk, room-1: “Meeting rooms are booked through the facilities page.” The model drafts “Rooms are booked at the front desk” and cites room-1 with the quote “at the front desk”. The citation names a real chunk, and the chunk does not contain the quote: an unsupported claim. A second draft cites room-7 for the same sentence: a dangling citation. Both are caught by the same few lines of code, and neither is caught by a prompt that says “cite your sources”.

The technique, restated from Build and Evaluate RAG Applications: score each chunk by the number of different non-stop words it shares with the question; take the top chunk; if its score is below a refusal threshold, refuse; otherwise run the citation check on what the model drafted. A second way to see it is a library desk: the librarian looks on the shelf (retrieval), says “we do not have that” when nothing matches well enough (refusal), and checks the page number you quote before you write it down (citation check).

The threshold is a trade. With a threshold of 2, a question sharing one word with its best chunk is refused even if that chunk holds the answer; with a threshold of 1 it is answered from a weak match. Neither is free, so the capstone records the threshold as a decision and counts what it refuses. Evidence that the problem is real comes from a paper on generative search engines, whose authors report that 51.5% of generated sentences were fully supported by citations and that 74.5% of citations supported their associated sentence (Liu, Zhang and Liang, Evaluating Verifiability in Generative Search Engines, retrieved 2026-10-04). Those are the paper’s measurements of the systems it audited, not a figure for yours; yours is whatever your check counts.

• Applied use. Retrieve, apply the threshold, draft, check every citation in code, and serve only what passes, or refuse.
• Real-world signal. A citation that cannot be checked by code, such as a bare “see the policy”, is a promise and not evidence.
• Practical consequence. A refusal is a cost paid on purpose: some answerable questions are refused, and the count of refusals belongs in the record so that someone can retune the threshold.
• Versus the alternatives. Asking the model to “only answer from the documents” is the distractor, because nothing then checks that it did. A passing citation check is also not proof that the answer is right: it shows that the quote is in the chunk, and says nothing about whether the quote answers the question or whether the chunk is out of date.

Bounded tools: your code decides

Take the support assistant’s request “where is order ORD-1001?”. The model replies with a tool call order_status with order_id set to ORD-1001. Anthropic’s documentation describes the division of work for a client tool: the model responds with a tool_use block, “Your code executes the operation and sends back a tool_result” (Anthropic, Tool use with Claude, retrieved 2026-10-04). The model asks; your application acts. That makes your code the place for the rules, and a short list covers the support assistant:

Rule 1  The name must be on the allow-list.               Rule 4  No more calls than the budget in one request.
Rule 2  The argument names must be exactly the tool's.    Rule 5  A consequential tool needs human approval.
Rule 3  The id must match the pattern over the whole string.   Then  Pass on only the exposed columns of what comes back.

Order matters because the first rule that applies is the one reported. A cancel call with a malformed id is reported as a bad argument and not as “needs approval”, so the person reading the log learns the real reason. A full-string match matters too: a pattern that only checks the start of the string accepts an id followed by stray characters, such as ORD-1001x.

The reason for each rule appears in OWASP’s entry on excessive agency, which names three root causes: “excessive functionality”, “excessive permissions” and “excessive autonomy”, and lists preventions that include minimising extensions, constraining permissions and requiring human approval before significant actions (OWASP GenAI, LLM06:2025 Excessive Agency, retrieved 2026-10-04). The allow-list and argument rules limit functionality, the exposed columns limit permissions, and the approval rule limits autonomy. A second picture is a reception desk with a printed menu: a visitor may ask for anything on the menu, a request for something off it is not improvised, and a request to enter a locked room needs a manager’s signature.

• Applied use. Put the checks in one function that runs before every tool call, and log its decision beside the call.
• Real-world signal. A dispatcher that falls back to the nearest tool name, or that passes through arguments it did not expect, is the shape of excessive functionality in code.
• Practical consequence. Exposing only named columns (the column-level grant of Taking AI to Production) means that a model tricked into asking for an email address cannot be handed one, because the column is not passed on to it.
• Versus the alternatives. A prompt that says “do not cancel without asking” is the distractor: it is a request to the model, and the model is the thing being checked. Giving the tool a broad database role and trusting the schema is the other, because a schema describes what a call should look like and says nothing of what the caller may reach.

Release evidence: repeated runs, a cost cap and a trace

Take a candidate release tested on 4 cases, each run 3 times, so a case can score 0 to 3 accepted replies. A written rule says: every case needs at least 2 of 3, and the total must be at least 11 of 12. Counts of 3, 3, 2, 3 total 11 and pass. Counts of 3, 3, 3, 1 total 10 and fail twice over. Repeating each case is what makes the rule fair to a component whose replies vary, which is the release gate taught in Taking AI to Production.

Two more rules cover what the gate cannot see. A cost rule: with prices of 2 and 10 dollars per million tokens, a span of 2000 input tokens and 300 output tokens costs 2000 × 2 + 300 × 10 = 7000 micro-dollars; the rule compares the candidate’s total with a cap, in whole numbers. A trace rule: every test case must appear in the trace, because a case with no span cannot be costed or explained afterwards. Tokens are the natural thing to record: the OpenTelemetry generative-AI conventions list the input and output token counts as recommended attributes, and mark message content, system instructions and tool definitions as opt-in (OpenTelemetry, Semantic conventions for generative client AI spans, retrieved 2026-10-04). A second picture: a pre-flight checklist where each line is yes or no and any “no” holds the flight, whatever else looks good.

• Applied use. Write the release rule, with its numbers, before the candidate is tested; run it over repeated results; report which rules failed, not only that something did.
• Real-world signal. A candidate that passes the gate and fails the cost cap is a normal outcome, and a release process that checks only quality will miss it.
• Practical consequence. The record that the rule reads (counts, cost, spans) is the same record that later shows a reviewer what the feature did, so keeping it is part of the deliverable.
• Versus the alternatives. “It looked good in the demo” is the distractor, because it is one run by one person. Another is a single overall pass rate, which can hide a case that fell from 3 to 1 while the total held steady.

Three situations you will meet, and what they mean for your design

Before the table, three situations:

1. A reviewer asks you to show where an answer’s wording came from.
2. The model proposes a call that no one planned for, such as a tool you did not offer.
3. A candidate beats the old release on quality but costs more per request than the agreed cap.

What you see	What it means	What to do instead
A reviewer wants provenance (situation 1)	A citation that cannot be checked in code	Serve only answers whose citations pass the check, and log the chunk id and quote
A call for an unplanned tool (situation 2)	The allow-list is the barrier in front of the tool	Deny it in code with a named reason, and log the attempt
Better quality, over the cap (situation 3)	A release rule failing on cost, not quality	Hold the release, report the failed rule, and change the prompt or the model, not the cap

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

What is a walking skeleton, and why start with one?

A tiny implementation of the system that performs a small end-to-end function and links the main components. Starting there means that every later step is added to a path that already runs, so a failure points at the step just added.

What does a refusal threshold decide, and what does it cost?

It decides whether the top chunk’s score is high enough to answer from. A higher threshold refuses some answerable questions; a lower one answers from weak matches. Either way the count of refusals belongs in the record.

What is the difference between a dangling citation and an unsupported claim?

A dangling citation names a chunk id that does not exist. An unsupported claim cites a chunk that exists but does not contain the quoted wording.

What does a passing citation check not prove?

It shows that the cited chunk exists and contains the quote. It does not show that the quote answers the question or that the chunk is current.

Who runs a client tool, the model or your application?

Your application. For a client tool the model responds with a structured call, and your code executes it and sends back the result, so your code is where the checks go.

Name the three root causes of excessive agency in OWASP's entry.

Excessive functionality, excessive permissions and excessive autonomy. The allow-list limits functionality, exposed columns limit permissions, and human approval limits autonomy.

Why does the order of the tool rules matter?

The first rule that applies is the one reported. A cancel call with a malformed id should be reported as a bad argument, so the log shows the real reason and not a later rule such as “needs approval”.

Why match a pattern over the whole string?

A pattern that checks only the start accepts text with extra characters after a valid prefix. A full-string match accepts only the exact shape.

How is the cost of a span computed, and what is a micro-dollar here?

Input tokens times the input price plus output tokens times the output price. With prices in dollars per million tokens, the result is in micro-dollars, so no division is needed.

Why does a release rule check trace coverage as well as quality?

A test case with no span cannot be costed or explained afterwards, so a candidate whose trace misses a case cannot be audited. The rule makes the record part of what must be true to ship.

Why can one candidate pass the gate and still be held?

The gate looks at accepted replies over repeated runs. A separate rule for cost or trace can still fail, and a release ships only when it fails none of its rules.

Model answer

The question: “You have to put a feature that answers from company documents and can look up an order into an existing system. What do you build first, what has to be true before it ships, and what evidence shows that it is?” This is what a hiring panel for a customer-facing engineering role asks to hear your order of work. A strong answer hits five beats, in this order:

1. Skeleton — a thin end-to-end path through the host system first, with a route, a trace and one test.
2. Ground — retrieval, a refusal threshold and a citation check in code.
3. Bound — tool calls checked by an allow-list, argument rules, a call budget, human approval and exposed columns.
4. Measure — a regression set run several times, with cost and a trace for each case.
5. Release — a written rule over those numbers, reporting which rule failed.

“I’d start with a walking skeleton: one route in the host system that runs end to end, writes a trace and passes one test, so every later step is added to something that works. Then I’d ground the answers: retrieve, refuse when the best match is below a threshold, and check every citation in code so that a dangling or unsupported one never reaches a user. Next I’d bound the tool: my code checks each call against an allow-list, exact argument names and patterns, a per-request budget and an approval rule, and it passes on only the columns the host’s owners agreed to expose. Then I’d measure: a regression set run three times per case, with token counts and cost in the trace. Finally I’d decide the release by a rule I wrote beforehand over those numbers, and report which rule failed if it fails, because that is the evidence a reviewer who wasn’t there can check.”

Missing a beat is a failure state. Without skeleton a failure cannot be placed to a step. Without ground an answer’s source cannot be checked. Without bound the model’s request decides what runs. Without measure there are no numbers to decide with. Without release the numbers are not turned into a decision.

Why this matters

• Reviewers read evidence, not effort. A capstone is judged by someone who was not there, so what counts is what a record lets them check.
• Controls live in code. Each promise in this chapter is kept by code that runs on every request, and a promise kept only by a prompt is a request to the thing being checked.
• A release is a decision, and decisions can be written down. Writing the rule before testing keeps the numbers from being argued after the fact.
• Talking to customers. Being able to say what the feature refuses, what it may touch and which rule held back a release answers the questions a customer’s reviewers ask.

After this chapter

The project for this chapter is Ground It, Bound It, Release It (project.md). You will run nine drafted answers through a refusal threshold and a citation check, fourteen proposed tool calls through a rule set, and four candidate releases through a release rule. You finish with an evidence file and three working functions.

This is the last chapter. Keep your evidence file: it is the record that lets someone who was not there check what you built and what the numbers show.

Hands-on Project — Ground It, Bound It, Release It

Objective

Check nine drafted answers against stored chunks, run fourteen proposed tool calls through a written rule set, and apply a release rule to four candidate releases, so that you can say what each control decides and which candidate ships.

What this project needs: Python 3 and its standard library only (re). No task needs an account, an API key, a network connection or any payment. The model’s drafts and tool calls are printed below, so every result can be checked exactly. All three tasks are required and are independent: each needs only the data and rules printed below.

Your evidence file. Keep one plain-text file, chapter11-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print. A list is comma-separated with no spaces and in the order stated; an empty list is none.

The data and the rules for all tasks

import re

STOP = {"the","a","an","is","are","of","to","for","do","i","my","can","how","what","when","in","on",
        "it","you","after","be","by","if","with","and","or"}
CHUNKS = {
    "ret-1": "Items may be returned within 30 days of delivery if they are unused.",
    "ret-2": "Refunds are paid to the original payment method within 5 business days of the return arriving.",
    "ship-1": "Standard delivery takes 3 to 5 business days. Express delivery takes 1 to 2 business days.",
    "ship-2": "Orders over 50 dollars ship free with standard delivery.",
    "warr-1": "Electronics carry a 365 day warranty against defects.",
}
THRESHOLD = 2
# (request id, question, cited chunk ids, quote the draft relies on)
DRAFTS = [
    ("q1", "Within how many days can items be returned?", ["ret-1"], "within 30 days of delivery"),
    ("q2", "When are refunds paid?", ["ret-2"], "within 5 business days"),
    ("q3", "How long does express delivery take?", ["ship-1"], "1 to 2 business days"),
    ("q4", "Is delivery free for large orders?", ["ship-2"], "over 100 dollars"),
    ("q5", "How long is the warranty on electronics?", ["warr-9"], "365 day warranty"),
    ("q6", "Do you sell gift wrapping?", [], ""),
    ("q7", "Can I return an unused item after delivery?", ["ret-2"], "within 30 days of delivery"),
    ("q8", "What does the electronics warranty cover?", ["warr-1"], "against defects"),
    ("q9", "Do you ship to Canada?", [], ""),
]

TOOLS = {"order_status": ["order_id"], "cancel_order": ["order_id"]}   # tool -> its exact argument names
NEEDS_APPROVAL = {"cancel_order"}
MAX_CALLS = 2
# (request id, tool name, arguments); calls of one request are listed in the order they were made
CALLS = [
    ("a1", "order_status", {"order_id": "ORD-1001"}),
    ("a2", "order_status", {"order_id": "ORD-10"}),
    ("a3", "delete_order", {"order_id": "ORD-2002"}),
    ("a4", "order_status", {"order_id": "ORD-3003", "include": "all"}),
    ("a5", "order_status", {"order_id": "ORD-4004"}),
    ("a5", "order_status", {"order_id": "ORD-4005"}),
    ("a5", "order_status", {"order_id": "ORD-4006"}),
    ("a6", "cancel_order", {"order_id": "ORD-5005"}),
    ("a7", "order_status", {}),
    ("a8", "cancel_order", {"order_id": "5005"}),
    ("a9", "order_status", {"order_id": "ORD-6006x"}),
    ("a10", "order_status", {"order_id": "ORD-7007"}),
    ("a10", "order_status", {"order_id": "ORD-7008"}),
    ("a10", "cancel_order", {"order_id": "ORD-7009"}),
]
HOST_ROW = {"order_id": "ORD-1001", "email": "pat@example.com", "status": "shipped",
            "card_last4": "4242", "carrier": "Northline"}
EXPOSED = ["order_id", "status", "carrier"]

CASES = ["k1", "k2", "k3", "k4", "k5", "k6"]          # each case was run 3 times
PRICE_IN, PRICE_OUT = 3, 15      # dollars per million tokens, so micro-dollars per token
COST_CAP_PER_CASE = 5000         # micro-dollars
# per candidate: accepted counts (out of 3) per case, and trace spans (case id, input tokens, output tokens)
RELEASES = {
    "v1": ([3, 3, 3, 3, 3, 2], [("k1", 900, 120), ("k2", 850, 100), ("k3", 1000, 140),
                                ("k4", 950, 110), ("k5", 900, 130), ("k6", 880, 125)]),
    "v2": ([3, 3, 3, 3, 3, 3], [("k1", 1500, 200), ("k2", 1500, 200), ("k3", 1500, 200),
                                ("k4", 1500, 200), ("k5", 1500, 200), ("k6", 1500, 200)]),
    "v3": ([3, 3, 1, 3, 3, 3], [("k1", 900, 120), ("k2", 850, 100), ("k3", 1000, 140),
                                ("k5", 900, 120), ("k6", 900, 120)]),
    "v4": ([3, 2, 2, 3, 2, 3], [("k1", 900, 120), ("k2", 850, 100), ("k3", 1000, 140),
                                ("k4", 950, 110), ("k5", 900, 130), ("k6", 880, 125)]),
}

Task 1 — Ground it

Words of a text are re.findall(r"[a-z0-9]+", text.lower()) with the words in STOP removed. The score of a chunk for a question is the number of different words they share. The top chunk is the one with the highest score; on a tie, the one listed first in CHUNKS. Write status(question, cites, quote), which returns the first rule that applies:

1. refused when the top chunk’s score is below THRESHOLD.
2. dangling when any cited id is not in CHUNKS.
3. uncited when cites is empty.
4. unsupported when, for any cited chunk, the lower-cased quote is not inside the lower-cased chunk text.
5. ok otherwise.

Run it over DRAFTS. Print the top chunk id of each question, the status of each, the ids that were refused, and the number of ok statuses over the number of requests that were not refused, as ok/served.

Add to your evidence file:

t1_top: <nine chunk ids>
t1_status: <nine statuses>
t1_refused: <request ids, or none>
t1_ok_served: <ok/served>

Output: a status function and four evidence lines.

Task 2 — Bound the tools

Write check_call(name, args, index), where index counts the earlier calls of the same request (0 for the first). It returns the first rule that applies:

1. unknown_tool when name is not in TOOLS.
2. bad_args when the argument names are not exactly the tool’s argument names (a missing or an extra one).
3. bad_args when order_id does not match ORD- followed by exactly four digits over the whole string (re.fullmatch).
4. budget when index is at least MAX_CALLS.
5. needs_approval when the tool is in NEEDS_APPROVAL.
6. allow otherwise.

Then write expose(row), which returns only the EXPOSED columns, in the order of EXPOSED. Run check_call over CALLS and expose over HOST_ROW.

Add to your evidence file:

t2_decisions: <fourteen decisions in order>
t2_allowed: <number of allow decisions>
t2_exposed: <column names returned by expose, in order>
t2_withheld: <columns of HOST_ROW not returned, in HOST_ROW order>

Output: two functions and four evidence lines.

Task 3 — Decide the release

The cost of a candidate is the sum over its spans of input_tokens * PRICE_IN + output_tokens * PRICE_OUT, in micro-dollars. A candidate fails a rule as follows, and ships only when it fails none:

• case: some case has fewer than 2 accepted replies of 3.
• total: the accepted replies over all cases total fewer than 16.
• cost: the cost is greater than COST_CAP_PER_CASE * len(CASES).
• trace: the set of case ids in the spans is not exactly the set in CASES.

Write release(counts, spans), which returns the failed rule names in the order case, total, cost, trace, and the cost. Run it over RELEASES.

Add to your evidence file:

t3_v1_fails: <failed rules, or none>
t3_v2_fails: <failed rules, or none>
t3_v3_fails: <failed rules, or none>
t3_v4_fails: <failed rules, or none>
t3_v1_cost: <micro-dollars>
t3_v2_cost: <micro-dollars>
t3_v3_cost: <micro-dollars>
t3_ship: <candidate ids that fail no rule, or none>

Output: a release function and eight evidence lines.

What carries forward

Keep chapter11-evidence.txt and your three functions status, check_call and release. This is the last chapter: your evidence file is the record that lets someone who was not there tell what you built and what the numbers show.