Chapter 2 — Cost and Latency as Design Constraints

Course: Getting Started with AI

Prerequisites: The ideas from Chapter 1, all of which are re-defined in the notes before they are used: a token is the unit a model reads and writes, the context window holds input and output together, and max_tokens is a ceiling on the length of a reply. You can read and run a short Python script. No account with a model provider and no API key are needed.

What the reader can do after this chapter:

• Calculate what one model request costs from its input tokens, its output tokens and a published per-million-token price list, and scale that figure to a stated daily volume.
• Estimate how long a reader waits for a reply from its time to first token and its output length, and say which change — fewer output tokens, fewer input tokens, or streaming — moves the wait the reader actually feels.
• Choose between interactive requests, a batch job and prompt caching for a described workload, and justify the choice with the cost and the wait it produces.

Core question this chapter answers: Before you build it, what will one request cost, how long will someone wait for it, and which design lever changes each of those two numbers?

Connections:

• Builds on: Chapter 1’s token counts — a request’s instruction, its input and its output ceiling, measured in tokens.
• Leads into: Chapter 3, which starts from the instruction you price in this chapter’s project and rewrites it as an interface with a contract: system and user roles, examples, and a defined output.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a number in an exact format, and each is marked by a rule printed with the question.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines. Every line’s accepted value, or the rule that decides it, follows from the prices and formulas printed in the project, so you can check your own file before you submit it.

Getting Started with AI — Chapter 2: Cost and Latency as Design Constraints

Before you start

Nothing to install for the reading. For the project you need Python 3 and its standard library only. You do not need an account with any model provider, an API key or any payment: every price the project uses is printed in it.

Keep one question in mind while you read: if this feature were used ten thousand times tomorrow, what would it cost, and how long would each person wait? By the end of the chapter you should be able to answer both from a request’s token counts and a price list, before anything is built.

Summary

A model request has two numbers a design review will ask you to defend: what it costs and how long someone waits for it. Both are driven by tokens, but not in the same way. Cost is input tokens times an input price plus output tokens times an output price, and on the price list this chapter uses an output token costs five times as much as an input token, so a short reply can still be a large share of the bill. Waiting time is dominated by generating the reply, one token after another, so cutting output tokens shortens the wait far more than cutting input tokens does, and streaming changes when the reader first sees text rather than when the reply is finished. Three levers then reshape the two numbers: batch processing trades waiting time for a lower price, prompt caching makes a repeated opening of a request cheaper and quicker to start, and model choice moves both. Knowing which lever moves which number is what lets you estimate a feature before you build it.

Key concepts

Term	Definition
Token	The unit a model reads and writes: a word, part of a word, a space plus a word, or a punctuation mark. Both prices and the context window are counted in tokens.
Input tokens	Everything sent to the model in a request — instructions, conversation history, documents, tool definitions — counted in tokens.
Output tokens	The tokens the model generates in its reply. max_tokens sets the most it may generate; the reply can be shorter.
Price per million tokens (MTok)	How usage-based model prices are quoted: a dollar amount for every million input tokens and a separate amount for every million output tokens.
Latency	The time it takes the model to process a prompt and generate an output.
Time to first token (TTFT)	The time from sending the request until the first token of the reply is generated.
Streaming	Receiving the reply in pieces as it is generated, instead of as one complete response at the end.
Batch processing	Submitting many requests together for asynchronous processing, at a discount, with results available later rather than immediately.
Prompt caching	Storing the processed opening part (the prefix) of a prompt so that later requests starting with exactly the same prefix are read from the cache at a lower price.
Cache write / cache read	A cache write stores a prefix the first time, at a price above the normal input price. A cache read reuses it, at a price far below the normal input price.

Explanations

The failure, before the explanation

A team builds a ticket-summary feature. In testing, they paste a ticket, get a good summary, and estimate the cost from the prompt they can see: a ticket of about 1,000 tokens at the input price, which on the price list below is $0.002 a request. The number is tiny and nobody worries.

In production the feature is used 20,000 times a day. Each request turns out to cost $0.015 — seven and a half times the estimate — and users complain that the summary “takes forever”. Three things were missing from the estimate:

1. A fixed instruction block of about 4,000 tokens is sent with every request, ahead of the ticket. Nobody counted it, because nobody types it.
2. The summaries average 500 tokens, and output tokens are priced higher than input tokens on the list they use.
3. Each summary is generated one token at a time, and the users stare at a spinner until the last token arrives.

None of these is a fault in the model. They are what cost and latency are made of. The rest of the chapter takes them one at a time, using that team’s request as the worked example: 4,000 tokens of fixed instructions, 1,000 tokens of ticket, 500 tokens of reply.

Cost: two prices, two token counts

Start with the bill, computed the way the provider computes it. One provider’s published price list (Anthropic, Pricing, retrieved 2026-10-04) quotes every model in dollars per million tokens, with separate columns for input and output. For the model this chapter uses as its example, the list shows:

Price item	Dollars per million tokens
Base input tokens	$2
Output tokens	$10
5-minute cache write	$2.50
Cache hit (read)	$0.20
Batch input / batch output	$1 / $5

The cost of one request is two multiplications and an addition:

cost = input_tokens × input_price ÷ 1,000,000 + output_tokens × output_price ÷ 1,000,000

For the worked example: 5,000 input tokens × $2 ÷ 1,000,000 = $0.010, and 500 output tokens × $10 ÷ 1,000,000 = $0.005. One request costs $0.015; at 20,000 requests a day that is $300 a day.

Look at the split. The reply is 500 of the 5,500 tokens — about 9% of the tokens — but it is a third of the cost. On that page’s model table, every row’s output price is five times its input price, so each output token weighs five times as much in the bill as an input token. An estimate made from the prompt alone misses this entirely.

A second way to picture it: a phone plan that charges for calls you make and, at five times the rate, for minutes you talk. Counting only how many calls you place tells you nothing about the bill if some calls run long. Requests are the calls; output tokens are the long minutes.

Two details change the count, and both come from Chapter 1’s idea that tokens, not words, are the unit:

• The tokenizer matters. The same provider notes that its newer models use a tokenizer that “produces approximately 30% more tokens for the same text” (Anthropic, Pricing). The same sentence can cost a different amount on a different model before the price per token is even compared.
• Hidden input counts. Anything sent in the request is input: the instruction block, earlier conversation turns, retrieved documents, tool definitions. The same price list notes that offering the model tools adds a tool-use system prompt of between 264 and 804 tokens per request, depending on the model and setting, before the tool definitions themselves are counted. If it is sent, it is billed.

Not every limit on a request is a price. Accounts also carry request-rate and spend limits, set separately by the provider (Anthropic, Rate limits); a design that is affordable per request can still be refused for sending too many requests too quickly.

Latency: where the waiting comes from

Now the spinner. Start with the measurements. One provider defines latency as “the time it takes for the model to process a prompt and generate an output”, and time to first token as the time “from when the prompt was sent” until “the first token of the response” is generated (Anthropic, Reducing latency, retrieved 2026-10-04).

Picture a printer receiving a long document. There is a pause while it takes in the job, then pages come out one by one. The pause depends a little on the size of the job; the total time depends mostly on how many pages come out. A model’s reply works the same way, because, as Chapter 1 showed, it is generated one token at a time. Another provider’s guide says it directly: “Generating tokens is almost always the highest latency step when using an LLM”, and gives the consequence in numbers: “cutting 50% of your output tokens may cut ~50% of your latency”, while “cutting 50% of your prompt may only result in a 1–5% latency improvement” (OpenAI, Latency optimization, retrieved 2026-10-04).

This chapter plans with a simple approximation built on that: total wait ≈ time to first token + output tokens ÷ output speed. It is a planning estimate this chapter uses, not a formula any provider publishes, and real speeds vary by model, load and request. For the worked example, assume a time to first token of 0.6 seconds and an output speed of 50 tokens per second — round teaching figures, not measurements of any model:

Change	Estimated wait
As built: 500 output tokens	0.6 + 500 ÷ 50 = 10.6 s
Ask for a 250-token summary	0.6 + 250 ÷ 50 = 5.6 s
Halve the instruction block, keep 500 output tokens	Still about 10.6 s — the guide puts input cuts at a 1–5% improvement

A second way to see it: a waiter’s time to bring a meal depends on the dishes they carry out, not on how long the order slip was. Trimming the order slip barely changes when dinner is served; ordering fewer dishes does.

Two consequences for design:

• Ask for less output when less is needed. The same latency guide recommends asking the model directly to be concise and setting max_tokens as a hard limit. But it also warns that a reply reaching max_tokens is “cut off, perhaps mid-sentence or mid-word” (Anthropic, Reducing latency). The ceiling protects the budget; asking for a shorter reply in the instruction is what produces a shorter, complete one.
• The model is a lever too. The same provider recommends choosing a faster model for speed-critical work, and its price list shows the model it calls its fastest at $1 input and $5 output per million tokens against $10 and $50 for its most expensive (Anthropic, Pricing). Whether the smaller model’s answers are good enough is a quality question that Build and Evaluate RAG Applications teaches you to measure.

Streaming: changing when the reader sees text, not when the reply ends

The ticket summary takes about 10.6 seconds to finish. With streaming, the reader sees the first words at about the time to first token — 0.6 seconds in the worked example — and then watches the rest appear. The last token still arrives at about 10.6 seconds.

That is the whole effect, and it is worth stating precisely: one provider calls streaming the approach that “cuts the waiting time to a second or less” (OpenAI, Latency optimization), and another says it improves “the perceived responsiveness of your application” (Anthropic, Reducing latency). Both describe the wait the reader feels. Neither says the reply finishes sooner.

A second picture: a progress bar on a file download does not make the download faster, but people tolerate a long download with a progress bar far better than a frozen screen. Streaming is the progress bar, made of the answer itself.

Streaming changes how your code reads the response, and three facts from one provider’s streaming documentation matter in practice (Anthropic, Streaming messages, retrieved 2026-10-04):

• The response arrives as a sequence of server-sent events — a message_start, content events, one or more message_delta events and a final message_stop — and the token counts in the message_delta usage field “are cumulative”. Read the final one for the request’s output count; do not add them up.
• Errors can arrive inside a stream that already started. The documentation shows an overloaded_error event that “would normally correspond to an HTTP 529 in a non-streaming context”. A stream that opened successfully is not yet a successful reply.
• For long replies streaming is not only cosmetic: the same documentation notes that for “requests with large max_tokens values” the SDK “requires streaming to avoid HTTP timeouts”.

Streaming suits a person reading the reply as it is written. A program that needs the complete reply before it can act — parsing a JSON object, for example — gains little from seeing the first half early.

Batch processing: trading the wait for a lower price

Some work has no one waiting for it. A team that labels last week’s 40,000 tickets for a monthly report due in three days needs the labels within three days, not within ten seconds.

For that shape of work, providers offer batch processing. One provider’s Message Batches API is described as suited “to tasks that do not require immediate responses, with most batches finishing in less than 1 hour while reducing costs by 50%” (Anthropic, Batch processing, retrieved 2026-10-04). Its documented limits are part of the design: a batch holds up to 100,000 requests or 256 MB, results are available “when all messages have completed or after 24 hours, whichever comes first”, and requests that have not been processed within 24 hours expire. Another provider documents the same trade: a “50% cost discount compared to synchronous APIs”, with each batch completing “within 24 hours (and often more quickly)” (OpenAI, Batch API, retrieved 2026-10-04).

On the worked example’s price list, batch input is $1 and batch output $5 per million tokens, so the $0.015 request costs $0.0075 in a batch. The latency is no longer seconds; it is “sometime within a day, often within an hour”, and the design must tolerate that.

A second picture: overnight post versus a courier. The courier is there in an hour and costs more; overnight post is cheaper and arrives tomorrow. Neither is better. The deadline decides.

Prompt caching: paying once for the part that never changes

Go back to the 4,000 tokens of fixed instructions sent with every ticket. Every request starts with the same 4,000 tokens, and every request pays to have them processed again.

Prompt caching stores the processed opening of a prompt — its prefix — so that a later request starting with exactly the same prefix can reuse it. One provider describes it as reducing “processing time and costs for repetitive tasks or prompts with consistent elements” (Anthropic, Prompt caching, retrieved 2026-10-04). The prices on the worked example’s list are a 5-minute cache write at $2.50 and a cache read at $0.20 per million tokens, against $2 for ordinary input; the documented multipliers are 1.25 times the base input price for a 5-minute write and 0.1 times for a read on most of that provider’s models (Anthropic, Pricing).

Run the numbers on the worked example:

Request	Instruction block (4,000)	Ticket (1,000)	Reply (500)	Total
No caching	$0.008	$0.002	$0.005	$0.015
First request, writes the cache	4,000 × $2.50 ÷ 1M = $0.010	$0.002	$0.005	$0.017
Later request, reads the cache	4,000 × $0.20 ÷ 1M = $0.0008	$0.002	$0.005	$0.0078

The first request costs a little more; each later one costs about half. The provider puts the break-even plainly: caching “pays off after one cache read for the 5-minute duration” (Anthropic, Pricing). It also helps the wait: the same documentation says that “you will generally see improved time-to-first-token for long documents” (Anthropic, Prompt caching).

A second picture: a coffee shop that grinds beans for each cup versus one that keeps a hopper of ground coffee for the morning rush. Grinding a full hopper costs a little more once; every cup after that is quicker and cheaper — as long as everyone wants the same coffee and the hopper is used before it goes stale.

Those two conditions are the rules caching runs on, and they are where designs go wrong:

• Exactly the same prefix. “Cache hits require 100% identical prompt segments” (Anthropic, Prompt caching). Another provider: “Cache reuse requires the entire rendered prefix to match” (OpenAI, Prompt caching, retrieved 2026-10-04). A timestamp or a user’s name at the top of the instruction block makes every prefix different, and nothing is ever read from the cache. That provider’s advice is to “Put stable developer instructions and shared reference material first” and to place timestamps and user-specific content “at the end rather than the beginning”.
• Used before it expires, and long enough to qualify. One provider’s default cache lifetime is 5 minutes, with a 1-hour option at twice the base input price for writes, and each model has a minimum cacheable length — 512, 1,024, 2,048 or 4,096 tokens depending on the model; shorter prompts “cannot be cached, even if marked” (Anthropic, Prompt caching). Another provider enables caching by default on supported models, with a 1,024-token minimum on its latest ones (OpenAI, Prompt caching).

You can check whether caching is working from the response. One provider reports cache_creation_input_tokens, cache_read_input_tokens and input_tokens separately, and their sum is the total input processed (Anthropic, Prompt caching); another reports usage.input_tokens_details.cached_tokens (OpenAI, Prompt caching). A cache read count that stays at zero across many identical-looking requests means the prefixes are not identical.

Three situations, and what they mean for your design

Before the table, three situations:

1. A support tool drafts a reply while an agent waits at their desk. Each request carries the same 6,000-token policy document before the customer’s message.
2. A data team classifies last week’s 40,000 tickets into categories. The results feed a report that is compiled three days from now.
3. A chat assistant answers questions about a contract. Users say the answer “hangs” and then appears all at once; the answers average 800 tokens.

What you see	Which number it moves	Lever to reach for
Same long prefix on every request (situation 1)	Cost of every request, and its time to first token	Prompt caching, with the stable document first and anything per-request after it
A large job with a deadline days away (situation 2)	Cost, at the price of a wait of up to a day	Batch processing, because the deadline is longer than the documented 24-hour window
A long reply a person reads (situation 3)	The wait the reader feels	Streaming for the feel; a shorter requested reply for the real total

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

How do you calculate the cost of one request from a per-million-token price list?

Multiply input tokens by the input price and output tokens by the output price, divide each by one million, and add the two. On the worked example, 5,000 input and 500 output tokens at $2 and $10 per million cost $0.010 + $0.005 = $0.015.

Why can a short reply be a large share of a request's cost?

Output tokens are priced separately and, on the list this chapter uses, at five times the input price. In the worked example the reply is about 9% of the tokens but a third of the cost.

What counts as input tokens in a request?

Everything sent: the fixed instruction block, earlier conversation turns, retrieved documents, tool definitions, and the user’s message. If it is in the request, it is billed, whether or not anyone typed it.

What is time to first token, and what is it not?

The time from sending the request until the first token of the reply is generated. It is not the total latency: the rest of the reply is generated after it, one token at a time.

Which cut shortens the wait more: half the output tokens or half the prompt?

Half the output tokens. One provider’s guide says that may cut about 50% of the latency, while halving the prompt may give only a 1–5% improvement, because generating tokens is almost always the slowest step.

Does streaming make a reply finish sooner?

No. It makes the first text appear at about the time to first token, so the reader waits less before seeing anything. The last token arrives at about the same time as without streaming.

A stream opened successfully. Is the reply guaranteed to complete?

No. One provider documents error events inside a stream, such as an overloaded_error that would be an HTTP 529 without streaming. Code reading a stream has to handle an error after text has started.

When is batch processing the right choice?

When nobody needs the answer within seconds and the work’s deadline is longer than the provider’s documented window — up to 24 hours on the two providers cited. “Most batches finish within an hour” is typical behaviour, not a promise to plan a deadline on. In exchange, the documented price is 50% lower.

What two conditions must hold for a prompt cache to be read?

The prefix must be exactly identical, up to and including the cached part, and the request must arrive before the cache expires — 5 minutes by default on one provider. The prefix must also meet the model’s minimum cacheable length.

Your cache read count stays at zero although every request "uses the same instructions". What do you check first?

Whether the prefix is really identical — look for a timestamp, a request id or a user’s name near the top of the instructions. Anything that changes per request belongs after the stable part.

Model answer

The question: “Before we build this feature, tell me what it will cost per day and how long users will wait — and what you would change if either number is too high.” This is the question a lead engineer or a customer’s architect asks in a design review. A strong answer hits five beats, in this order:

1. Count — the request’s input and output tokens, including everything sent with every request.
2. Price — input and output priced separately, then scaled by the daily volume.
3. Wait — time to first token plus the time to generate the output, which dominates.
4. Levers — which lever moves which number: caching for a repeated prefix, batch for work whose deadline is longer than the batch window, a shorter requested output for the wait, streaming for the wait the reader feels.
5. Verify — how you would confirm the estimate from the response’s own usage fields once it runs.

“Each request sends about 4,000 tokens of fixed instructions plus a 1,000-token ticket, and the summaries come back at about 500 tokens. On the price list we’re using that’s $2 per million input and $10 per million output, so $0.010 for input and $0.005 for output — a cent and a half a request, about $300 a day at 20,000 requests. The reply is under a tenth of the tokens but a third of the cost, because output is priced five times higher. For the wait, generation dominates: assuming 50 tokens a second, a 500-token summary takes about ten seconds after the first token. If cost is the problem, the fixed instructions are the same on every request, so caching them should bring later requests to roughly half price, as long as nothing per-request sits in that prefix. If the wait is the problem, I’d ask for a shorter summary, which shortens the real total, and stream it, which cuts the time before users see anything. Once it’s live I’d check the usage fields — input, output and cache read counts — to confirm the estimate rather than trust it.”

Missing a beat is a failure state. Without count the estimate leaves out the fixed instructions that nobody types. Without price split by direction it misses that output is the expensive side. Without wait the answer has no latency figure at all. Without levers it describes the problem and offers no fix. Without verify it is an estimate nobody will ever correct — and that is the beat reviewers most often find missing.

Why this matters

• Design reviews. “It’s cheap” is not an estimate. A per-request figure with the input and output split shown, scaled to a stated volume, is something a reviewer can check and a budget owner can approve.
• Customer conversations. A customer who asks why the assistant “feels slow” is usually asking about the wait before the first words appear, which streaming addresses, or about long replies, which shorter output addresses. Knowing which one saves a week of the wrong optimisation.
• Choosing an architecture. Interactive, batch and cached designs are not interchangeable; the deadline and the shape of the prompt decide which one fits, and the wrong one can cost double for the same work.
• Operations. Usage fields in every response are the meter. Logging them is how a team notices a cache that stopped hitting or a reply length that drifted upward before the bill does.

After this chapter

The project for this chapter is Price and Time One Request (project.md). You will write a cost calculator from a printed price list, estimate waits for different output lengths, and finish by pricing one real request from your own work — its instruction, its input and its output ceiling.

Chapter 3 starts from the instruction you priced in the last task. It treats that instruction as an interface: it splits it into system and user roles, adds examples, and defines its output, and you will see what each change does to the token counts you measured here.

Hands-on Project — Price and Time One Request

Objective

Write a cost calculator and a waiting-time estimator from a printed price list and a stated planning formula, use them to compare interactive, cached and batch designs, and finish by pricing one real request from your own work, so that you can put a cost and a wait on a feature before anyone builds it.

What this project needs: Python 3. Tasks 1 and 2 use only the standard library. Task 3 also needs the tiktoken package (pip install tiktoken, which needs an internet connection). No task needs an account with a model provider, an API key or any payment. All three tasks are required.

The price list for every task, in dollars per million tokens. These figures are copied from one provider’s published list for one model (Anthropic, Pricing); use them exactly as printed here even if the live list has changed since.

Price item	Dollars per million tokens
Input	2.00
Output	10.00
5-minute cache write	2.50
Cache read	0.20
Batch input	1.00
Batch output	5.00

Your evidence file. Keep one plain-text file, chapter2-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print.

Task 1 — Build a cost calculator

Write one Python file with these functions. Each takes token counts and returns dollars.

1. plain_cost(input_tokens, output_tokens) — input at the input price plus output at the output price, each divided by 1,000,000.
2. cached_cost(prefix_tokens, other_input_tokens, output_tokens, hit) — the prefix is priced at the cache read price when hit is true and at the 5-minute cache write price when it is false; other input at the input price; output at the output price.
3. batch_cost(input_tokens, output_tokens) — input and output at the two batch prices.

Then, for the notes’ worked example — 4,000 tokens of fixed instructions, 1,000 tokens of ticket, 500 tokens of reply — print each value below rounded to 4 decimal places (to 2 for the daily figure, to 3 for the share):

• the plain cost of one request, with all 5,000 input tokens at the input price;
• the cached cost when the 4,000-token prefix is written (hit false);
• the cached cost when the 4,000-token prefix is read (hit true);
• the batch cost of one request;
• the plain cost of 20,000 requests a day;
• the share of the plain cost that comes from output: output cost ÷ plain cost.

Before you print those, sanity-check your functions: plain_cost(1_000_000, 1_000_000) must return 12.0 (2.00 + 10.00) and batch_cost(1_000_000, 1_000_000) must return 6.0 (1.00 + 5.00). Stop there: three functions and six printed numbers, not a pricing library.

Add to your evidence file:

t1_plain: <4 decimal places>
t1_cache_write: <4 decimal places>
t1_cache_read: <4 decimal places>
t1_batch: <4 decimal places>
t1_daily_plain: <2 decimal places>
t1_output_share: <3 decimal places>

Output: one Python file with three functions, and six evidence lines.

Task 2 — Estimate the wait

This task uses the planning approximation from the notes: total wait = time to first token + output tokens ÷ output speed. It holds the time to first token fixed, so it shows only the effect of output length; it is a planning estimate, not a measurement of any model. Use a time to first token of 0.6 seconds and an output speed of 50 tokens per second.

1. Write wait(output_tokens, ttft=0.6, speed=50) returning seconds.
2. Print the wait, to 1 decimal place, for replies of 500 and 250 output tokens.
3. Print the time at which a streamed reply of 500 tokens shows its first text, to 1 decimal place. In this approximation that is the time to first token.
4. Answer two design questions as single words, using only the source text printed here:
   • OpenAI, Latency optimization (retrieved 2026-10-04): “cutting 50% of your output tokens may cut ~50% of your latency” and “cutting 50% of your prompt may only result in a 1–5% latency improvement”. Which halving shortens the total wait more — halving the output tokens or halving the input tokens? Write output or input.
   • Anthropic, Batch processing (retrieved 2026-10-04): batch results can be accessed “when all messages have completed or after 24 hours, whichever comes first”, and “most batches” finish within 1 hour. A job must finish within 12 hours. Does this text guarantee a batch finishes in time? Write yes or no.

Add to your evidence file:

t2_wait_500: <seconds, 1 decimal place>
t2_wait_250: <seconds, 1 decimal place>
t2_first_text_streamed: <seconds, 1 decimal place>
t2_bigger_lever: <output or input>
t2_batch_guaranteed_12h: <yes or no>

Output: one function and five evidence lines.

Task 3 — Price one real request

Pick one request you would really send to a model in your work: a short instruction (a few lines at most) plus one piece of input it works on — a log excerpt, a ticket or a diff. One request, one input, about one screen of text at most. If you kept instruction.txt and input.txt from an earlier project, you may reuse them.

1. Save the instruction in instruction.txt and the input in input.txt.

2. Count the tokens in each file with the tiktoken package and its cl100k_base encoding:

   import tiktoken
   
   enc = tiktoken.get_encoding("cl100k_base")
   for name in ("instruction.txt", "input.txt"):
       with open(name, encoding="utf-8") as f:
           print(name, len(enc.encode(f.read())))

   This counts tokens with one tokenizer. Tokenizers differ between models: Anthropic, Pricing says its newer models’ tokenizer “produces approximately 30% more tokens for the same text” than the previous one. Treat the result as an estimate of this request’s size.

3. Choose an output ceiling: the most tokens you would allow the reply to use, as a whole number.

4. Price the request as if the reply used the whole ceiling — the most it could cost: (instruction tokens + input tokens) × 2.00 ÷ 1,000,000 + ceiling × 10.00 ÷ 1,000,000. Print it to 6 decimal places, then multiply the unrounded cost by 1,000 and print the cost of a thousand such requests to 2 decimal places. (If you wrote Task 1’s plain_cost, it computes exactly this.)

5. Estimate the wait for a reply that uses the whole ceiling: 0.6 + ceiling ÷ 50 seconds, printed to 1 decimal place. (Task 2’s wait computes exactly this.)

Add to your evidence file:

t3_instruction_tokens: <tokens in instruction.txt>
t3_input_tokens: <tokens in input.txt>
t3_output_ceiling: <your chosen ceiling>
t3_max_cost: <dollars, 6 decimal places>
t3_cost_per_1000: <dollars, 2 decimal places>
t3_max_wait: <seconds, 1 decimal place>

Output: two text files that make up one real request, and six evidence lines that put a price and a wait on it.

What carries forward

Keep instruction.txt, input.txt and chapter2-evidence.txt. The next chapter opens with the instruction you priced in Task 3: it rewrites that instruction as an interface — a system part and a user part, with examples and a defined output — and you will compare the new token count and cost with the figures you recorded here.