Chapter 8 — Capstone: Design an End-to-End AI System

Course: Taking AI to Production

Prerequisites: You can read and run a short Python script. The earlier material each taught one control, and the notes restate each in a line before it is used: retrieval and reranking (Chapter 2), limits and degradation (Chapter 3), data freshness (in Build and Evaluate RAG Applications), cost caps and routing (Chapter 4), agent identity and least privilege, approval gates and tenant isolation (all in Building AI Agents) and audit records (Chapter 7). Requirement, coverage, cap, trade-off and nearest cheaper option are defined in the notes before they are used.

What the reader can do after this chapter:

• Check a draft design against a list of requirements and two caps, and say which requirements are uncovered and which cap is broken.
• Find, among all combinations of options, the design that covers every requirement and stays within both caps.
• Write down, for each decision, the nearest cheaper option with its saving, what it would stop meeting and which cap it would break.

Core question this chapter answers: Given an organisation’s requirements, how do you assemble retrieval, agency, approval, identity, isolation, audit and cost into one design, and show what each choice traded away?

Connections:

• Builds on: Chapter 7’s audit-gaps file. The project opens by classifying the gaps in your audit-gaps.tsv and asking which control in the design addresses them.
• Leads into: Chapter 9, which opens with the lines of your tradeoffs.tsv, one challengeable decision each, and asks how to record and defend them.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines and two tab-separated files. Every accepted value follows from what the task prints.

Taking AI to Production — Chapter 8: Capstone, Design an End-to-End AI System

Before you start

Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library. You do not need an account with any cloud or model provider or an API key: the requirements, options and caps the project uses are printed in the project file.

Keep one question in mind while you read: if a reviewer points at any part of my design and asks why it is there, which requirement can I name?

Summary

A complete design is a set of decisions, each one answering a stated requirement, and not a list of components that sound modern. This chapter starts from the requirements, maps each to a control taught earlier (retrieval, approval, identity, isolation, audit, cost routing), and checks that every requirement is covered by at least one chosen option. Costs and waiting times add up across the stages of a request, so a choice that meets its own requirement can still break a cap on the total. For each decision the design records the nearest cheaper option, what it would save, what it would stop meeting and which cap it would break, and it also lists what is left uncovered. The project turns this into a check you can run.

Key concepts

Term	Definition
Requirement	A statement of what the system must achieve or respect, written so that you can tell whether a design meets it, such as “a payment over 200 waits for a person’s approval”.
Control	A mechanism in the design that meets a requirement, such as an approval gate or a separate index for each tenant.
Decision	One choice in the design among several options, for example how retrieval is done.
Option	One of the alternatives for a decision, with its cost, its added waiting time and the requirements it meets.
Coverage	A requirement is covered when at least one chosen option is stated to meet it. A requirement no chosen option meets is uncovered.
Cap	A hard limit on a total across the whole design, such as cost per 1,000 requests or seconds of waiting added to one reply.
Cost per 1,000 requests	The spend for a thousand requests. It is not the cost per resolved item, which divides spend by the useful results.
Trade-off	What a choice gives up in return for what it gains: a saving paid for with a requirement left unmet, or a requirement met at the price of a cap.
Nearest cheaper option	Among the options that cost less than the chosen one, the one that costs most. It is the alternative a reviewer is most likely to propose.
Retrieval, rerank	Retrieval looks facts up in your documents and places them in the prompt. Reranking scores the candidates again with another model and keeps the best few.
Approval gate	A point where an action does not run until a person has approved it.
Least privilege, delegation	Least privilege grants only the permissions a task needs. Delegation means an agent acts for a user while keeping its own identity, so a record can name both.
Tenant isolation	Controls that use the tenant of a request to limit which data it can reach.
Hash chain, anchor	A hash chain makes each record’s stored hash depend on the one before it, so an alteration shows. An anchor is a copy of the latest hash kept where the log’s writer cannot change it.
Model routing	Sending each request to the model that is good enough for it, so easy requests go to a smaller, cheaper model.

Explanations

The failure, before the explanation

A retailer’s team presents a design for a returns assistant. The slide lists a retrieval index over the returns policy, an agent that can issue refunds, a content filter, single sign-on, a monthly cost alert and application logs. Each item sounds sensible. At launch, an agent refunds an order far above the usual size and nobody approved it, because no requirement said refunds above some amount need a person. A busy period pushes spend past what the finance team expected, and the alert arrives after the money is spent. Staff in one store see policy notes written for another store. When a customer disputes a refund, the logs show a request and the word “success”, not who the agent acted for.

Nothing in the design was broken. Nothing in it was tied to a requirement either, so nobody could say what it was meant to prevent, which gaps remained, or what each part cost.

Start from requirements, and check coverage

Two ways to see it. A building inspector works from a checklist: each line names something the building must satisfy, and a missing tick is a finding. A recipe works the same way in reverse: you do not start from the pans in the cupboard, you start from the dish and list what it needs. A design that starts from its components has the cupboard and no dish.

The method is to write each requirement as a sentence you could test, then write beside it the option or options in the design that meet it. For the returns assistant: “A refund above a stated amount waits for a person” is met by an approval gate. “A store’s staff read only that store’s documents” is met by tenant isolation. “A dispute can be answered from the records” is met by audit records that name the user and the agent. A requirement with an empty cell is uncovered, and an option with no requirement beside it is a cost that nobody asked for.

• Applied use. Build a table with one row per requirement and one column per decision, and mark where an option meets a requirement. Count the empty rows before anything else.
• Signal. Words such as “must”, “at most”, “only”, “before” and “for whom” in a scenario are requirements. A stem that lists technologies and asks which to add is asking you to find the requirement first.
• Practical consequence. An uncovered requirement is a launch risk you chose without noticing. An option that meets nothing still costs money and adds a stage to the request path.
• Versus the distractor. The tempting answer adds a fashionable component. Ask which stated requirement it meets, and discard it if the answer is none.

Compose the controls from the earlier chapters

An airport is a good picture. Check-in confirms who you are, security limits what you carry, the boarding gate stops you boarding the wrong flight, and the cameras and tickets let someone reconstruct events later. No single layer does the job, and each one answers a different question. A second picture is a bank branch: the teller checks identity, a manager approves a large withdrawal, the vault keeps each customer’s box apart, and the ledger records the steps.

The controls of this course map onto the same set of questions. One line for each, with the chapter it came from:

Question the design must answer	Control	Taught in
How much of the work does the model do alone?	A single call, retrieval, reranking, a workflow or an agent, each adding stages and model calls	Chapter 2
What happens when capacity or a provider’s limit runs out?	Limits, backoff, queues and degradation modes	Chapter 3
Where do the documents live and how stale may they be?	A source of truth, ingestion and a stated staleness limit	in Build and Evaluate RAG Applications
What may be spent, and counted against whom?	Forecasts, caps and routing to a cheaper model for easy requests	Chapter 4
Who is the agent, and what may it hold?	Its own identity, least privilege and delegation	in Building AI Agents
Which actions wait for a person?	An approval rule and an approval gate	in Building AI Agents
What can a mistake or a compromise reach?	Account boundaries, tenant isolation and sandboxes	in Building AI Agents
How will you prove afterwards what happened?	Records with a trace and sequence number, a hash chain, an anchor and denial records	Chapter 7

The earlier chapters each treated one row. The capstone’s job is the combination, where two controls touch. Isolation and retrieval touch: a tenant filter has to apply before documents are returned to the prompt, or the isolation is on paper only. Identity and audit touch: a record can name the user only if the agent acts by delegation and not through a shared credential. Approval and cost touch: a person in the path adds waiting time.

• Applied use. Draw the request path from arrival to answer, and mark on it where each control sits. Where two controls sit next to each other, ask whether one depends on the other.
• Signal. A stem that mixes personal data, payments and “prove later” is asking for several rows of the table at once. A stem about one of them usually wants one control.
• Practical consequence. Controls added at the wrong point can be bypassed: a tenant check applied after retrieval has already put another tenant’s text in the prompt.
• Versus the distractor. The tempting answer is the strongest single control, such as a very strict approval step, with the other questions left open.

Caps add up across the stages

Two examples. A courier delivery has legs: collection, sorting, transport and the last mile. A customer’s promised arrival is the sum of the legs, and adding a security check to one leg lengthens the total. A household budget is similar: rent, food and travel each look affordable, and the total is what the account feels. For an AI request, each stage adds seconds and adds cost, so a design has a total cost per 1,000 requests and a total of seconds added to one reply, and each can have a cap.

A small case. A library’s lending assistant has three stages. Retrieval costs 6 and adds 2 seconds. A privacy filter costs 3 and adds 1 second. An approval step for fines costs 1 and adds 3 seconds. The totals are 10 and 6 seconds. If the cap on waiting is 5, no single stage looks large and the design is still over. The fix is to change a stage, for example to a cheaper filter that adds less time, or to take the approval step off the path for small fines, and not to argue that each stage is small.

Two sources make the same point from the pillars of one cloud framework. Its cost page lists “setting hard spending limits can prevent a workload from scaling to meet legitimate demand” among the ways saving money can reduce resilience. Its security page says inspection controls such as firewalls and content safety filters “sit in the flows that they secure”, so those flows are subject to additional verification, “which adds latency to requests”. It also says high-fidelity security monitoring and audit data that provide broad coverage increase storage costs (Microsoft, Azure Well-Architected Framework, cost and security trade-off pages). The project’s cost and seconds are invented numbers, and its rule that they simply add is a simplification.

Keep two measures apart. The cost per 1,000 requests is a spend per thousand calls. The cost per resolved item divides spend by the useful results, and the two diverge when many requests fail or are retried. A cap on one is not a cap on the other.

• Applied use. List each stage’s cost and seconds, sum them, and compare each sum with its cap. Do this for every candidate design, not once for the final one.
• Signal. “At most”, “within”, “while the user waits” and “budget” point toward caps. A requirement met by a component is a coverage question. A total that is too large is a cap question, and the two have different fixes.
• Practical consequence. A design can cover every requirement and still be invalid. Adding a control raises the totals, so the controls that meet the requirements set a floor the caps must leave room for.
• Versus the distractor. The tempting answer is to drop a requirement when the total is over. Look first for another option that meets the same requirement with a smaller total.

Make each trade-off explicit

Two ways to see it. A shopper who picks a more expensive suitcase has a reason: it survived the airline, or it was the only one under the weight limit. A restaurant bill split fairly lists what each person had, so a dispute can be settled from the list. A design that states only what it chose leaves a reviewer to guess what was refused.

For each decision, record four things. The option chosen. The nearest cheaper option, since that is the one a reviewer will propose. What switching to it would save. What it would cost you: the requirements it stops meeting, and any cap that the switched design would break. Take an invented case. A lending assistant uses a per-branch index at cost 9 and 0 seconds. The nearest cheaper option is a shared index with a branch filter at cost 4 and 2 seconds. Swapping saves 5. Both meet the requirement that one branch cannot read another’s documents, so nothing is lost on coverage, but the swap adds 2 seconds, and if the design was at 5 seconds of a cap of 6, it breaks the cap. The record says so in one line: saves 5, loses no requirement, breaks the wait cap.

The cost page of the cloud framework warns about the opposite habit. For cost optimization, “it’s important to avoid focusing on a cheaper solution”, because “choices that focus only on minimizing spending can undermine your workload’s business goals and reputation”. It also says that sharing resources to increase density can leave co-located resources sharing a workload identity, with less meaningful audit trails (Microsoft, Cost Optimization tradeoffs). Saving money is one of the questions, and the record shows which others it moved.

• Applied use. Write one line per decision: chosen option, nearest cheaper option, saving, requirements lost, caps broken. A reviewer can then challenge the line and not a vague sense of the design.
• Signal. “Why not the cheaper one?”, “what do we lose?” and “justify the cost” ask for this record. A line that loses no requirement and breaks no cap is a signal that the chosen option may be too dear.
• Practical consequence. The record turns a later argument into arithmetic. If a cap changes, the lines that said “breaks the cap” are the ones to reopen.
• Versus the distractor. The tempting answer is a general claim that the chosen option is “more robust”. Name the requirement or the cap, or the claim cannot be checked.

Say what the design leaves open

Two examples. A pilot’s checklist ends with the items deliberately deferred and why, not only with the items done. A builder’s handover lists the work left for the owner. In a design, the same list goes with the table: the requirements no chosen option meets, the assumptions the numbers rest on, and the risks a control reduces and does not remove. Chapter 7 gave one: a sequence number reveals a hole in the middle of a trace and not a trace that stops early.

• Applied use. End the design with a short list: uncovered requirements, assumptions (for example, that costs add), and known limits of the controls chosen.
• Signal. “What are the risks?” and “what could go wrong with this design?” ask for this list, and an answer of “nothing” is a weak one.
• Practical consequence. A listed gap can be accepted, scheduled or removed by a change. An unlisted gap is found by an incident.
• Versus the distractor. The tempting answer is a design with no stated gaps. It reads as complete and cannot be checked.

Three situations you will meet, and what they mean for your design

1. A reviewer asks you to show that each stated requirement is met, and your design is a diagram of components.
2. A design covers every requirement, and its total waiting time is one second over the cap.
3. A colleague proposes a cheaper option for one decision and asks why you did not take it.

What you see	What to do	Why
A diagram with no link to requirements (situation 1)	Write each requirement, and the option that meets it, in a table, then list the empty rows	Coverage is a statement you can check
Coverage complete, wait one second over (situation 2)	Look for another option, in any decision, that meets the same requirement with fewer seconds, and recompute the total	A cap failure is fixed by changing a stage and not by dropping a requirement
A cheaper option proposed (situation 3)	Show the saving, the requirement lost and the cap broken, from the record	The record turns the challenge into a comparison of numbers

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

Why start a design from requirements and not from components?

A component list cannot show what each part is for, which requirements are unmet or what each part costs. Requirements written as testable sentences let you map each to an option, count the uncovered ones and discard an option that meets nothing.

What does it mean for a requirement to be covered?

At least one chosen option is stated to meet it. A requirement with no such option is uncovered, and it is listed in the design as a gap instead of left to be found later.

Name the eight questions the earlier controls answer.

How much the model does alone, what happens at the limit of capacity, where documents live and how stale they may be, what may be spent and counted against whom, who the agent is and what it holds, which actions wait for a person, what a mistake or compromise can reach, and how to prove afterwards what happened.

Give a case where two controls depend on each other.

Tenant isolation and retrieval: the tenant filter has to apply before documents reach the prompt. Identity and audit: a record can name the user only if the agent acts by delegation and not through a shared credential.

Why can a design that covers every requirement still be invalid?

Costs and seconds add across stages, so the chosen options can total more than a cap. Coverage and caps are separate checks: coverage asks whether each requirement has an option, and a cap asks whether the sum is within a limit.

A design is over its wait cap. What do you try first?

Look for another option, in any decision, that meets the same requirement with fewer added seconds, and recompute the total. Dropping a requirement is a decision for the owner of the requirement and not a repair.

What is the difference between cost per 1,000 requests and cost per resolved item?

The first is spend for a thousand calls. The second divides spend by useful results, so failed or repeated requests raise it without changing the first. A cap on one is not a cap on the other.

What is the nearest cheaper option, and why record it?

Among the options that cost less than the chosen one, the one that costs most. It is the alternative a reviewer is most likely to propose, so the record shows its saving, the requirements it would lose and the caps it would break.

What does the cloud framework's cost page warn about?

That choices focused only on minimizing spending can undermine the workload’s business goals and reputation. A cheaper option is judged against the requirements and caps, not by price alone.

What belongs in the closing list of a design?

The uncovered requirements, the assumptions the numbers rest on (for example that costs add) and the known limits of the chosen controls, such as a sequence number not revealing a trace that stops early.

Model answer

The question: “Here is an organisation’s problem. Walk me through the AI system you would design, and show me what you traded away.” An architect meets it in a design review or a hiring loop. A strong answer hits five beats, in this order:

1. Requirements — testable statements of what the system must do and respect, written first.
2. Controls — an option for each requirement, drawn from retrieval, approval, identity, isolation, audit and routing, with the points where two touch.
3. Caps — the totals of cost and waiting time checked against their limits.
4. Trade-offs — for each decision the nearest cheaper option, its saving, what it loses and what it breaks.
5. Open items — uncovered requirements, assumptions and known limits.

“I would begin by writing the requirements as sentences I can test, such as a payment above a stated amount waiting for a person, one tenant not reading another’s documents, and the records naming both the user and the agent. For each one I would choose a control: an approval gate, tenant isolation applied before retrieval, delegation with least privilege, and audit records with a chain and an anchor. I would check where two controls depend on each other, for example that the tenant filter comes before the documents reach the prompt. Then I would add up cost per thousand requests and the seconds added to a reply, and compare them with the caps, because a design that covers every requirement can still be too slow or too dear, and I would change a stage before dropping a requirement. For each decision I would write the nearest cheaper option, what it saves, which requirement it would stop meeting and which cap it would break, so that a challenge becomes a comparison. I would close with what remains open: any uncovered requirement, the assumption that costs add, and the limits of the controls.”

Missing a beat is a failure state. Without requirements the controls have nothing to answer to. Without controls the requirements are wishes. Without caps a design can be correct and unusable. Without trade-offs the reviewer’s cheaper option has no answer. Without open items the design claims a completeness it cannot show.

Why this matters

• Design reviews. A reviewer who asks “why this and not that” wants a requirement and a number, and a design with a table of both answers faster than one with a diagram.
• Hiring loops. A capstone-style question asks for a whole system, and the answer is judged on whether each part is tied to a need and the trade-offs are stated.
• Customer conversations. A customer who asks what the assistant can do on its own gets a clearer answer from a design that lists the approval rule and the isolation boundary than from one that lists products.
• Budget and capacity planning. Caps decide which options are available at all, so a team that sums them early finds an over-total design before it is built.

After this chapter

The project for this chapter is Design Under Constraints (project.md). You will check a draft design against requirements and caps, find the one valid design among 729 combinations, and write design.tsv and tradeoffs.tsv.

Chapter 9 opens with your tradeoffs.tsv. Each line is a decision a reviewer could challenge, and that chapter asks how to write each one down as a decision record and defend it.

Hands-on Project — Design Under Constraints

Objective

Check a draft design against stated requirements and caps, find the one design that meets them, and write down what each choice trades away, so that a complete design is something you can defend with numbers.

What this project needs: Python 3 and its standard library only. No task needs a cloud account, a model, an API key or an internet connection. Everything is printed below and every check is an exact comparison. All three tasks are required. Task 1 can be done at any point, and Task 3 builds on the design that Task 2 produces, so do Task 2 before Task 3.

Your evidence file. Keep one plain-text file, chapter8-evidence.txt, of key: value lines with the key in lower case exactly as printed. Write what your code printed, not what you expected. Write a list comma-separated with no spaces, in the order given, and an empty list as none.

The data and the rules for all tasks

The case is an insurer’s claims assistant. Staff ask it questions about policy documents, and it can issue small goodwill payments. Copy this into your Python file.

# The numbers seen for each trace in Chapter 7's project:
SEEN = {"tr1": [1, 2, 3, 5, 6, 9], "tr2": [1, 2, 3], "tr3": [2, 3, 4]}

REQUIREMENTS = {
    "R1": "An answer cites the document version it used",
    "R2": "A payment over 200 waits for a person's approval",
    "R3": "A request from one tenant cannot read another tenant's documents",
    "R4": "Records name both the user and the agent that acted",
    "R5": "Refusals and actions are both recorded",
    "R6": "A change to the records shows when they are checked",
    "R7": "The agent holds only the permissions its task needs",
    "R8": "Hard questions are answered by the large model",
}
CAP_COST = 50   # at most 50 cost units per 1,000 requests, summed over all six decisions
CAP_WAIT = 8    # at most 8 seconds added to one reply, summed over all six decisions

# decision -> option -> (cost per 1,000 requests, seconds added, requirements it meets)
OPTIONS = {
    "D1": {"A": (5, 1, set()),       "B": (12, 2, {"R1"}),       "C": (20, 4, {"R1"})},        # retrieval
    "D2": {"A": (0, 0, set()),       "B": (6, 2, {"R2"}),        "C": (9, 3, {"R2"})},         # approval
    "D3": {"A": (1, 0, set()),       "B": (2, 0, {"R7"}),        "C": (4, 1, {"R4", "R7"})},   # identity
    "D4": {"A": (2, 0, set()),       "B": (5, 2, {"R3"}),        "C": (11, 0, {"R3"})},        # isolation
    "D5": {"A": (1, 0, set()),       "B": (3, 0, {"R5"}),        "C": (7, 1, {"R5", "R6"})},   # audit
    "D6": {"A": (14, 3, {"R8"}),     "B": (8, 2, {"R8"}),        "C": (4, 1, set())},          # model routing
}
DRAFT = {"D1": "C", "D2": "A", "D3": "B", "D4": "B", "D5": "B", "D6": "A"}

A design picks one option for each of the six decisions. Its cost and its wait are the sums of the chosen options’ costs and seconds. A requirement is covered when at least one chosen option lists it in its meets set. A design is valid when all eight requirements are covered, the cost is at most CAP_COST and the wait is at most CAP_WAIT. Cost here is per 1,000 requests, and the rule that costs and seconds simply add is this case’s simplification.

Task 1 — Check the carry-in and the draft

Carry-in: a gap is a number from 1 up to the highest number seen in a trace that is not in its list. A gap is leading when it is smaller than the lowest number seen in that trace, and middle otherwise. Classify the gaps in SEEN. Then evaluate DRAFT: cost, wait, and the requirements it does not cover.

t1_gap_total: <number of gaps across the three traces>
t1_leading_gaps: <number of leading gaps>
t1_middle_gaps: <number of middle gaps>
t1_draft_cost: <cost of DRAFT>
t1_draft_wait: <wait of DRAFT>
t1_draft_uncovered: <requirement ids not covered by DRAFT, in order, comma-separated, or none>
t1_draft_valid: <true or false>

Output: one Python file with evaluate(design), and seven evidence lines.

Task 2 — Find the valid design

Try every combination of one option per decision (729 in all). Count the combinations that cover all eight requirements, the ones that cover all eight and also keep the cost within CAP_COST, and the ones that are valid. Write design.tsv: one line per decision in order, with the decision and the option of the cheapest valid design, separated by a single tab character.

t2_covering: <combinations covering all eight requirements>
t2_covering_within_cost: <combinations covering all eight with cost within CAP_COST>
t2_valid: <valid combinations>
t2_design: <the options of the cheapest valid design in order D1 to D6, comma-separated>
t2_cost: <its cost>
t2_wait: <its wait>

Output: design.tsv of six lines, and six evidence lines.

Task 3 — Write down what each choice trades away

Using the design from Task 2, take each decision in turn. Its nearest cheaper option is the option with the highest cost among those that cost less than the chosen one. Build a variant of the design with only that option swapped. Write tradeoffs.tsv: one line per decision in order, with five tab-separated fields: the decision, the nearest cheaper option, the saving (chosen cost minus that option’s cost), the requirements the chosen option meets and the nearest cheaper one does not (sorted, comma-separated, or -), and the caps the variant breaks (cost, wait or cost,wait, or -). A cap is broken when the variant’s total is above it.

t3_total_saving: <sum of the six savings>
t3_losses_total: <total number of requirement ids across the fourth fields>
t3_cap_blocked: <decisions whose variant breaks a cap, in order, comma-separated, or none>

Output: tradeoffs.tsv of six lines, and three evidence lines.

What carries forward

Keep design.tsv, tradeoffs.tsv and chapter8-evidence.txt. Chapter 9 opens with your tradeoffs.tsv. Each line is a decision somebody could challenge, because a cheaper option exists. That chapter asks how to record each decision so that a person who was not there can follow it, and how to defend it.