Chapter 4 — Cost Architecture

Course: Taking AI to Production

Prerequisites: You can read and run a short Python script. From earlier material you need three ideas, each restated in a sentence in the notes before it is used: a token is the unit of text a model reads and writes and is what usage is priced in, an index is rebuilt from a source by a job that embeds chunks (the ingestion plan in Build and Evaluate RAG Applications), and a provider limits how fast a caller may use a model. Forecast, unit price, cap, quota, prompt prefix, batch, routing and unit cost are defined in the notes before they are used.

What the reader can do after this chapter:

• Compute, from stated volumes, token counts and unit prices, what a model-backed feature costs per request and per day, and which feature dominates the total.
• Work out the saving from caching a stable prompt prefix, from sending work to a batch interface and from routing easy requests to a smaller model, and say which lever saves most for a given workload.
• Set a spending cap in the request path, convert it to a request limit, and defend the design in a review by naming its assumptions and what would change them.

Core question this chapter answers: How do you estimate what an AI feature will cost before it is built, make the cost controlled by the design and not by someone watching a dashboard, and defend the numbers when someone challenges them?

Connections:

• Builds on: the ingestion plan in Build and Evaluate RAG Applications. Its upsert lines are chunks that must be embedded again, which is one place spend arises. This chapter prices model calls in general.
• Leads into: the first governance chapter in Building AI Agents, which asks what an agent acts as. It opens with the budget-rules.tsv this chapter’s project writes, and asks which caller a cap can be counted against.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question, so you can check that your answer has the right shape before you submit it.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines and one tab-separated file. Every accepted value, or the rule that decides it, follows from what the task prints.

Taking AI to Production — Chapter 4: Cost Architecture

Before you start

Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library. You do not need an account with any provider or an API key: the volumes, token counts and prices the project uses are printed in the project file, and the prices are invented teaching figures, not any provider’s price.

Keep one question in mind while you read: if the spend doubles tomorrow, what in the design stops it, and what merely tells someone afterwards?

Summary

The cost of a model-backed feature can be forecast before it exists, because a call is commonly priced per input and output token, and a forecast needs the volume, the tokens per request and the unit prices. A forecast is a claim to be checked, so it is stated with its assumptions and a high case. Controlling spend is a separate act: an alert or a dashboard reports spend after the usage, while a cap in the request path or a bound on output length changes what can happen. Three levers cut the cost of the same work: reusing a stable prompt prefix, moving work that can wait to a batch interface, and sending easy requests to a smaller model, and the best one depends on the shape of the workload. A cost design that survives a review states a unit cost, the assumptions behind it, the controls that hold it, and the evidence that would change it.

Key concepts

Term	Definition
Token	The unit of text a model reads and writes. Providers commonly count usage and price calls in tokens, and often price input tokens and output tokens differently.
Unit price	The price of a stated quantity, such as cost units per 1,000 input tokens. The project uses invented cost units.
Forecast	An estimate of spend before it is incurred: requests, multiplied by tokens per request, multiplied by unit price. It holds as far as its assumptions do.
Cost per request	What one call costs: input tokens times the input price plus output tokens times the output price.
Unit cost	Spend divided by a useful result, for example the cost per resolved ticket. It ties the cost to what the system delivers.
Cap	A limit on spend or usage that the system itself enforces, before the next request is served, by refusing or degrading requests.
Alert	A message that spend or usage has passed, or is forecast to pass, a threshold. It informs a person, and it may arrive after the usage it reports.
Prompt prefix	The start of a prompt. Prompt caching reuses work when requests share the same prefix, so stable text goes first.
Batch interface	An asynchronous interface that takes groups of requests and returns results later, and may be priced lower than a synchronous call (the cited provider’s guide states 50% lower).
Routing	Sending each request to the model that is good enough for it, so that easy requests go to a smaller, cheaper model.
Quota	A limit that a provider or an organisation sets on how much a caller may use, for example a number of tokens.
Sensitivity	How much a forecast changes when one assumption changes. It shows which assumption to measure first.

Explanations

The failure, before the explanation

Here is a message from a finance partner, after a feature has launched.

Message: “The assistant’s spend is three times the figure in the plan. Nobody told us. The dashboard has a line, and the alert email went to a shared inbox. What did the plan assume?”

The plan assumed a number of requests and a prompt of a certain length. In this constructed case both were wrong: usage was higher, and the prompt had grown as people added instructions and examples. The team had an estimate and a dashboard. It had no statement of the assumptions, so nobody could tell which one had moved, and no cap, so nothing stopped the spend while the email waited. Two separate things went missing, a forecast that could be checked and a control that did not depend on a person reading in time.

Forecasting spend before it is incurred

Take a support assistant. Each request reads 1,200 input tokens and writes 250 output tokens. With invented prices of 3 cost units per 1,000 input tokens and 12 per 1,000 output tokens, one request costs 1.2 × 3 + 0.25 × 12 = 3.6 + 3.0 = 6.6 cost units. At 5,000 requests per day that is 33,000 cost units per day. The formula has three inputs, volume, tokens in and out per request, and unit prices, and each is an assumption that can be measured.

A second way to see it is a restaurant’s cost per dish: ingredients per plate times the price of the ingredients, times plates sold. Nobody argues about the formula. The arguments are about plates sold and the price of ingredients, which are the estimates.

Where you use it. Write the forecast as a small table: requests per day, input tokens, output tokens and the model, per feature. State a low, expected and high case for the volume, so the high case shows what the design costs if the forecast is wrong. Use the provider’s current price list for real work, and say when you read it.

What a review listens for. “What are the assumptions, and which one matters most?” A forecast that gives a total without its inputs cannot be challenged and so cannot be trusted. Sensitivity is the useful part: if doubling the output length moves the total more than doubling the requests, measure output length first.

What changes in operations. The project’s prices treat input and output tokens separately, and a price list may do the same, so a forecast keeps the two counts apart and reads the price list for the model in use. The forecast also needs a re-check whenever the prompt changes.

Versus the neighbour it gets confused with. A forecast is not a budget. A forecast predicts what will be spent given assumptions, and a budget is what the organisation allows. A design needs both, and the gap between them is what the controls are for.

Control by structure, not by vigilance

Return to the message. The alert existed, and it reached a person after the spend. The cloud provider’s budgets documentation says this can happen: “there can be a delay between when you incur a charge and when you receive a notification”, and “you might incur additional costs or usage that exceed your budget notification threshold before AWS Budgets can notify you” (AWS, Managing your costs with AWS Budgets). An alert is a useful signal, and it describes the past.

A cap in the request path works differently. Before each request is served, the system checks usage against an allowance and refuses or degrades the request when it is reached. A bound on output length is a smaller version of the same idea: one provider’s API reference describes max_tokens as “the maximum number of tokens to generate before stopping”, and notes that its models “may stop before reaching this maximum” (Anthropic, Messages API reference). It is a ceiling on one request’s output cost, and the model may stop sooner.

A second way to see it is a household’s electricity. An electricity bill tells you what you used. A circuit breaker stops a fault when it happens. You want both, and you would not rely on the bill to protect the wiring.

Where you use it. Place a cap per feature, with a limit expressed in requests per day derived from the allowed spend and the cost per request, plus a ceiling on output tokens per request. Decide what happens at the cap: refuse, or degrade to a cheaper model or a shorter answer. Keep alerts, set below the cap, as the signal that a cap is approaching.

What a review listens for. “What stops the spend if the forecast is wrong?” The answer “we get an alert” says that a person has to be fast. The answer “a cap in the request path, and an alert as the early signal” says that the design holds when no one is looking.

What changes in operations. A cap needs a decision about who is affected when it trips, and a way to raise it quickly for a good reason, such as a launch. It also needs the spend to be attributable to a caller, a feature or an agent; a cap on a pool of shared credentials cannot say which feature used the allowance, and the first governance chapter in Building AI Agents comes back to that.

Versus the neighbour it gets confused with. A rate limit restricts how fast you may call; a quota, as defined above, may be set by the provider or by your organisation. A provider-set limit can bound spend as a side effect, but it is not a statement of what you allow. A cap you set expresses your allowance.

Three levers that cut the cost of the same work

Take a workload that sends the same 3,000-token policy text at the start of each request. Three levers apply, and each fits a different shape of workload.

Reuse a stable prefix (prompt caching). One provider’s guide says “prompt caching reuses work when requests share the same prompt prefix”, that cached tokens are “discounted up to 95%”, and that the advice is to place “stable developer instructions and shared reference material first” and “changing content after the breakpoint” (OpenAI, Prompt caching guide). It also states a minimum prompt length for caching to apply, so a short prompt may not benefit. The practical consequence is about order: a prompt that begins with the changing question has no stable prefix to match.

Move work that can wait to a batch interface. The same provider’s batch guide describes its own asynchronous groups of requests “with 50% lower costs” (other providers’ batch pricing may differ), a completion window within which results come back, and rate limits that are “separate from existing per-model rate limits” (OpenAI, Batch API guide). It fits work where nobody is waiting, the batch shape from Chapter 1: summaries prepared ahead of need, a back-catalogue to classify, an ingestion job that embeds chunks. It does not fit a screen with a person looking at it.

Route easy requests to a smaller model. A smaller model may cost less per token, which is a comparison to check against the current price list for the models in question. The chapter’s reasoning, which is a design choice to measure and not a vendor claim, is that many requests in a feature are easy, so a classifier or a rule can send them to a smaller model and keep the larger one for the hard ones. The risk is quality: the saving is real when the smaller model is good enough, and that is established with an evaluation set (Getting Started with AI), not assumed.

A second way to see the three levers is a courier: the stable prefix is a parcel already packed and reused, batch is the slower, cheaper post, and routing is a bicycle for a short trip and a van for a long one.

Where you use it. Measure each lever alone on the forecast table, then combine. Levers can overlap, so savings do not simply add: batching a feature that is also routed to a smaller model halves a smaller cost.

What a review listens for. “Which lever saves most for this workload, and what does it cost to adopt?” Caching needs prompts restructured and a stable prefix long enough to qualify. Batch needs a feature that tolerates delay. Routing needs a quality check.

What changes in operations. Each lever adds a moving part to watch: a cache hit rate, a queue of batch jobs, a router’s error rate. A saving that is forecast is checked against the invoice.

Versus the neighbour it gets confused with. Fine-tuning a smaller model (Getting Started with AI) is a fourth route to a lower per-request cost, with its own one-off cost. Cutting the prompt is cheaper still when it does not hurt quality, and it is the first thing to try.

Defending the number in a review

Suppose a lead engineer asks, “Why is this affordable?” An answer that offers a total per day invites the reply “per day compared with what?”. A stronger answer is a unit cost: “6.6 cost units per request at 1,200 input and 250 output tokens, with 70% of tickets resolved without a human, so about 9.4 (6.6 / 0.7) per resolved ticket, against the stated cost of a human-handled one.” Each part can be challenged, which makes the claim stronger.

A second way to see it is a unit price on a menu. A restaurant that says “we spent 40,000 last week” has told you nothing, and one that says “a plate costs 6 and sells for 14” has told you what to check.

Where you use it. Write the cost per useful result, the assumptions in a short list, the controls that hold it, and the one or two measurements that would change the design, such as a higher-than-forecast volume or a prompt that has doubled in length.

What a review listens for. “What would make you change your mind?” Naming the evidence is the difference between a position and a preference.

What changes in operations. The assumptions list is what gets re-measured after launch. A cost report that tracks the assumptions, volume, tokens per request and unit price, can say which one moved when spend does.

Versus the neighbour it gets confused with. A price per token comparison between providers is a part of the forecast, and it does not say what a task costs: a cheaper model that needs two attempts may cost more per resolved ticket.

Three situations you will meet, and what they mean for your design

Before the table, three situations:

1. A feature’s prompt starts with the customer’s message and then a long block of fixed instructions, and the invoice is higher than the price list suggests.
2. A team runs a job that summarises a back-catalogue through the same interface as the live chat, and both draw on the same quota.
3. A feature has an alert at 80% of its allowance. The traffic spikes, and the allowance is gone before anyone opens the email.

What you see	What it means	What to do
Fixed text after the changing part (situation 1)	There is no stable prefix to reuse	Move the stable text to the start and the changing text after it, then check the cache hit rate
Work nobody waits for on the live interface (situation 2)	It pays the full price and competes with users for limits	Move it to a batch interface, which has a lower price and its own limits
An alert that arrives after the spend (situation 3)	The control is a signal and not a limit	Add a cap in the request path, with a decision for what happens when it trips, and keep the alert as the early signal

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

What three inputs does a spend forecast need?

The volume of requests, the tokens read and written per request, and the unit price of each kind of token. Multiply them: requests, times tokens per request, times unit price. Each input is an assumption that should be stated and later measured.

Why state a low, expected and high case?

Volume and prompt length are assumptions that can turn out wrong. A high case shows what the design costs if the forecast is wrong, and it tells you what cap to set.

Why can an output limit not be relied on to set the cost of a reply?

It is a ceiling and not a target. One provider’s reference says the model may stop before reaching the maximum, so it bounds the worst case for one reply without fixing the length or the cost.

What does the budgets documentation say about an alert and the spend it reports?

There can be a delay between incurring a charge and being notified, and spend or usage may exceed the threshold before the notification arrives. An alert reports the past, so it is not a limit.

What is the difference between an alert and a cap?

An alert informs a person and may arrive after the spend. A cap is checked by the system before each request and refuses or degrades it at the allowance. Use both: the cap holds the line and the alert warns that it is near.

What makes a prompt cache hit possible, and what follows for prompt design?

A later request sharing the same prefix as an earlier one. Put stable instructions and reference text first and the changing content after, so that the prefix matches. A prompt that starts with the changing part has nothing to match.

What does a batch interface trade for its lower price?

Waiting. The guide describes asynchronous groups of requests at 50% lower cost, with a completion window and separate rate limits, which suits work where nobody is waiting and not an interactive screen (the 50% figure is that provider’s).

When is routing to a smaller model a saving, and when is it not?

When the smaller model is good enough for the requests sent to it, which an evaluation set shows. If its quality is too low, retries and complaints can cost more than the tokens saved.

Why do the savings of the three levers not simply add up?

They act on the same requests. A feature that is routed to a smaller model and then batched has a smaller cost to halve, so the combined saving is less than the sum. Measure each alone, then the combination.

What is a unit cost, and why is it a stronger claim than a total?

Spend divided by a useful result, such as the cost per resolved ticket. Each part (cost per request, the assumptions and the rate of useful results) can be checked, and it ties the spend to what the system delivers.

Model answer

The question: “How would you make sure this AI feature stays within its cost, and how would you defend the numbers in a review?” This is the question a lead engineer or a finance partner asks before a launch, and an interviewer for a cloud AI architecture role asks to hear a design and not a total. A strong answer hits five beats, in this order:

1. Forecast — state the formula (requests, tokens in and out, unit prices), its assumptions and a high case.
2. Cap — name a control in the request path that holds the allowance, plus an output ceiling, with a decision for what happens when it trips.
3. Alert — keep an alert below the cap as the early signal, and say that it reports spend after the usage.
4. Levers — name which of prefix reuse, batch and routing fit this workload, and what each costs to adopt.
5. Evidence — give a unit cost, the assumptions behind it, and what measurement would change the design.

“I’d start by writing the forecast as requests per day times tokens in and out per request times the unit prices, and I’d show a low, expected and high case, because volume and prompt length are assumptions that may prove wrong. I wouldn’t rely on someone watching a dashboard. I’d put a cap per feature in the request path, derived from the allowance and the cost per request, with a ceiling on output tokens, and decide in advance whether it refuses or degrades. I’d keep an alert below the cap as the early signal, knowing that billing alerts can arrive after the usage. For the cost itself I’d look at three levers: put the stable text first in the prompt so a prefix can be reused, send work nobody is waiting for to a batch interface, and route easy requests to a smaller model once an evaluation set shows it is good enough, measuring each alone because they overlap. And I’d defend the number as a cost per resolved ticket, with the assumptions listed and the two measurements that would make me change the design.”

Missing a beat is a failure state. Without forecast the answer has no number to defend. Without cap it trusts a person to be fast. Without alert nobody learns that the cap is near. Without levers it accepts the first price it computed. Without evidence it is a total, and a reviewer is left to ask “compared with what?”.

Why this matters

• Design reviews ask for it. A proposal with no stated assumptions cannot be challenged, so a reviewer either rejects it or approves it on trust, and a forecast with a cap gets a faster yes.
• Money follows the design. The order of a prompt, the interface used for work that can wait and the model that receives easy requests change the invoice without changing the product.
• Surprises are a design failure, not bad luck. A cap and an early alert turn a campaign that doubles traffic into a decision, not an incident.
• Talking to finance. An engineer who gives a cost per resolved ticket, the assumptions and the controls is speaking the language the finance partner is checking.

After this chapter

The project for this chapter is Forecast the Spend and Set the Caps (project.md). You will compute the cost of five features from stated volumes and prices, measure what each of three levers saves, turn spending allowances into request limits, and write budget-rules.tsv.

The first governance chapter in Building AI Agents opens with the caps in your file. Each cap is counted against some caller, and a cap that cannot say who the caller was cannot be enforced or explained. It asks what an agent acts as when it spends and when it reads data.

Hands-on Project — Forecast the Spend and Set the Caps

Objective

Forecast what five model-backed features cost per request and per day from stated volumes and prices, measure what three cost levers would save, and turn spending allowances into request limits in a rules file, so that a cost claim rests on stated numbers and a cap that the design enforces.

What this project needs: Python 3 and its standard library only. No task needs an account with a provider, an API key or any payment. The prices below are invented teaching figures and are not the price of any provider’s model. Every check is an exact comparison. All three tasks are required.

Your evidence file. Keep one plain-text file, chapter4-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print. Where a value is a list, write its items comma-separated with no spaces, in the order given. Use exact arithmetic (Python’s fractions.Fraction, or whole numbers scaled by 1,000) so that totals are not off by a rounding error.

The data and the rules for all tasks

Copy this into your Python file. Cost units are an invented currency. PRICES gives, for each model, the cost units per 1,000 input tokens and per 1,000 output tokens. Each feature is (name, requests_per_day, input_tokens, output_tokens, model, batch_ok, small_ok, stable_pct): the volume, the tokens read and written per request, the model it uses, whether its work could wait for a batch interface, whether the smaller model would be good enough for it, and the percentage of its input tokens that form a stable prefix (text that is the same from call to call, placed first).

PRICES = {"small": (1, 4), "large": (10, 40)}
FEATURES = [
    ("chat-assist", 4000, 1500, 300, "large", False, False, 60),
    ("doc-summaries", 1000, 6000, 500, "large", True, True, 0),
    ("ticket-triage", 20000, 400, 20, "small", False, False, 50),
    ("report-draft", 300, 3000, 1500, "large", True, False, 20),
    ("faq-bot", 8000, 800, 150, "small", False, False, 75),
]
CAPS = {"chat-assist": 150000, "doc-summaries": 80000, "ticket-triage": 15000,
        "report-draft": 50000, "faq-bot": 15000}

The cost rule. One request on model m costs input_tokens / 1000 × input_price + output_tokens / 1000 × output_price. The cost per day is that times requests_per_day. The baseline is every feature on its own model with no lever.

The three levers. Each is measured alone against the baseline unless a task says otherwise.

• Route: a feature with small_ok true is priced on the small model.
• Cache: stable_pct percent of a request’s input tokens are billed at 10% of the input price; the rest of the input tokens and all output tokens are billed in full.
• Batch: a feature with batch_ok true has its request cost halved.

Combined means: route first, then cache, then batch, on each feature.

Task 1 — Forecast the baseline

Write request_cost(feature) for the baseline and compute the cost per request, the cost per day for each feature, the total per day, and the feature with the largest cost per day with its share of the total as a percentage.

Add to your evidence file:

t1_request_chat: <cost units per request for chat-assist>
t1_request_triage: <cost units per request for ticket-triage>
t1_total_day: <total cost units per day, whole number>
t1_top: <name of the feature with the largest cost per day>
t1_top_share_pct: <its share of the total, as a percentage with one decimal place>

Output: one Python file with request_cost, and five evidence lines.

Task 2 — Measure the levers

This task needs only the data and the rules above. For each of the three levers alone, compute the total cost per day for all five features and the saving against the baseline total. Then compute the combined total.

Add to your evidence file:

t2_save_route: <saving per day from route alone, whole number>
t2_save_cache: <saving per day from cache alone, whole number>
t2_save_batch: <saving per day from batch alone, whole number>
t2_best: <the lever with the largest saving: route, cache or batch>
t2_total_combined: <total cost per day with all three levers, whole number>

Output: the arithmetic, and five evidence lines.

Task 3 — Turn allowances into request limits

This task needs only the data and the rules above, with the baseline costs.

• A feature’s limit is the largest whole number of requests per day that keeps its baseline cost per day at or below its cap: the cap divided by the baseline cost per request, rounded down.
• A feature breaches a scenario when its baseline cost per day is greater than its cap. The forecast scenario uses the requests in FEATURES. The high scenario multiplies each feature’s requests by 3/2.

Write budget-rules.tsv: one line per feature, in FEATURES order, with three fields separated by single tab characters: the feature name, its cap and its limit.

Add to your evidence file:

t3_limit_report: <the limit for report-draft>
t3_forecast_breaches: <number of features that breach the forecast scenario>
t3_high_breaches: <names of the features that breach the high scenario, in FEATURES order>

Output: a budget-rules.tsv of five lines, and three evidence lines.

What carries forward

Keep budget-rules.tsv and chapter4-evidence.txt. The first governance chapter in Building AI Agents opens with the caps in your file: each is counted against some caller, and a cap that cannot say who the caller was cannot be enforced or explained. That chapter asks what an agent acts as when it spends.