Chapter 1 — AI Workload Shapes and Where They Run

Course: Taking AI to Production

Prerequisites: You can read and run a short Python script. From earlier material you need three ideas, each restated in a sentence in the notes before it is used: a token is the unit of text a model reads and writes and is what usage is counted and priced in, a provider limits how fast an organisation may call a model, and an agent’s tools can be reviewed for what they reach and what they may do (the tool review in Building AI Agents). Workload shape, batch, interactive, managed inference, reserved capacity, self-hosted inference and break-even are defined in the notes before they are used.

What the reader can do after this chapter:

• Classify a workload as interactive or batch from who is waiting and how soon the result is needed.
• Compute, from stated prices and capacities, the cheapest of on-demand, reserved and self-hosted inference at a given daily volume.
• Name what each place to run a model commits a team to operationally, and defend a choice in a design review by naming the requirement that would change it.

Core question this chapter answers: Given a workload that calls a model, how do you decide whether it is interactive or batch and whether to run it on managed on-demand inference, reserved capacity or a model you host yourself, and what does that choice commit you to operating?

Connections:

• Builds on: the review of an agent’s tools in Building AI Agents, which asked what a model-driven system can reach. This chapter asks where the model calls themselves run and what that costs to keep working.
• Leads into: Chapter 2, which gives the shapes retrieval and agentic systems take. It opens with the interactive workloads in the placement.tsv this chapter’s project writes.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question, so you can check that your answer has the right shape before you submit it.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines and one tab-separated file. Every accepted value, or the rule that decides it, follows from what the task prints.

Taking AI to Production — Chapter 1: AI Workload Shapes and Where They Run

Before you start

Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library. You do not need an account with any cloud or model provider or an API key: every workload, price and capacity the project uses is printed in the project file, and the prices are invented teaching figures, not any provider’s price.

Keep one question in mind while you read: who is waiting for this answer, and who will be running the machine that produces it?

Summary

A workload that calls a model has a shape before it has a technology: either a person or a calling system is waiting for the answer (interactive), or nobody is and a result is needed by a deadline (batch). The shape decides which places can serve it. Providers publish batch interfaces that complete work within a stated window at a stated discount and draw on a separate pool of limits, while interactive traffic needs an answer now. Interactive traffic can run on managed on-demand inference, on reserved capacity bought at a fixed cost, or on a model the team hosts and operates itself, and each choice moves cost from the request to the commitment and moves operational work onto the team. A defensible design names the shape, prices the options at the expected volume, and says what the team has taken on.

Key concepts

Term	Definition
Token	The unit of text a model reads and writes. Providers count usage, limits and prices in tokens.
Workload shape	How a workload calls a model: whether someone is waiting for the answer and how soon a result is needed. It decides which places can serve the workload.
Interactive workload	A workload where a person or a calling system is waiting for the answer, so it must come back quickly.
Batch workload	A workload where nobody is waiting and a result is needed by a deadline, so the requests can be submitted together and collected later.
Managed on-demand inference	Calling a model that a provider hosts and runs, sending requests as needed and paying for what is used. This chapter uses “on-demand” for that form of buying.
Reserved capacity	A fixed amount of model throughput bought from a provider at a fixed cost whether or not it is fully used, for example Amazon Bedrock’s Provisioned Throughput.
Self-hosted inference	Running a model on machines the team operates, usually with an open serving library such as vLLM. The team owns capacity, scaling and failures.
Rate limit	A cap a provider puts on how much an organisation may send in a period, for example requests per minute or tokens per minute. A request over the cap gets an error (HTTP 429).
Operational commitment	The work and money a choice binds a team to keep doing: paying for unused capacity, running machines, watching limits.
Break-even volume	The daily volume at which a fixed-cost option costs no more than paying per token. Below it the per-token option is cheaper.
Deadline	The latest moment a result is still useful. A batch workload has one and no one waiting.

Explanations

The failure, before the explanation

A team runs a customer chat assistant on one provider account. Late in the day it also starts a job that summarises a large archive of documents, using the same account and the same limits. The summaries are needed only by the next morning’s review. Part-way through the job the chat users start seeing errors, and the logs show the provider answering with rate-limit errors (HTTP 429, “too many requests”).

Nothing is broken. The two workloads share one pool of per-minute limits, and the job used most of it. One provider’s documentation notes that “unsuccessful requests contribute to your per-minute limit, so continuously resending a request won’t work” (OpenAI, Rate limits). The same page points non-urgent work to the provider’s batch interface “without impacting your synchronous request rate limits”. The fix was a property of the workload, not of the model: the summary job had a shape that did not need an answer now, and it was sent down the interactive path anyway.

Workload shape: who is waiting

Two examples first. A user types a question and watches a spinner: someone is waiting, so the answer has to come back while they are still there. A nightly job re-labels yesterday’s support tickets and writes the labels to a table that is read the next day: nobody is waiting, and what matters is that the table is full by morning. A second way to see it is the difference between a counter and a catering order. A customer at a counter needs the food in minutes, and the kitchen is staffed for that. A catering order is placed with a delivery deadline, and the kitchen is free to cook it whenever the counter is quiet.

The sorting rule is short: if a person or a calling system is waiting, the workload is interactive; if nobody is waiting and a result is needed only by a deadline, it is batch. The deadline matters as much as the waiting. A job nobody watches but that must finish this afternoon is not safely batch if the batch interface you would use only promises completion within a longer window.

Providers publish such interfaces and the promises differ, so read the stated one. OpenAI’s batch guide describes a “clear 24-hour turnaround time”, batches completing “within 24 hours (and often more quickly)”, and a “50% cost discount compared to synchronous APIs”. It says batch runs in “a separate pool of significantly higher rate limits” that “will not consume tokens from your standard per-model rate limits” (OpenAI, Batch API). Anthropic’s Message Batches page says “most batches” finish “in less than 1 hour”, cuts cost by 50%, and says batches “expire if processing does not complete within 24 hours”; requests that expire are not billed (Anthropic, Batch processing). Both pages state a 24-hour window and a 50% saving. Check the current page of the provider you use before relying on either figure.

• Applied use. Sort each workload by asking “who is waiting?” and “when is the result useful until?”. The first splits interactive from batch; the second splits batch from a job that merely feels background.
• Signal. Wording such as “nightly”, “overnight”, “backfill”, “needed for tomorrow’s review” and “no user is waiting” points toward batch; “user is waiting”, “before the payment is approved” and “inside the request” point toward interactive.
• Practical consequence. A batch workload can leave the interactive limits alone and cost less; an interactive one cannot. Mixing them on one account, as in the failure above, lets the cheap-to-delay work consume the capacity the waiting users need.
• Versus the distractor. The tempting wrong answer is to call any large job batch. Size is not the test: a very large job that must finish within the afternoon cannot rely on a window that is stated as up to 24 hours.

Three places to run interactive traffic

Take a chat assistant that must answer in seconds. It can run in three places, and each is easiest to picture as a way of getting somewhere. Managed on-demand is a taxi: you call it when you need it, pay for the ride, and own nothing, but you are one customer among many and the operator sets how many rides you may take at once. Reserved capacity is a leased car: you pay a fixed amount whether or not you drive, and in return a vehicle is yours. Self-hosted is owning the car: you buy, fuel, insure and repair it, and no one else decides how it is used.

Managed on-demand means the provider hosts and runs the model and you call it. The provider’s limits apply. Anthropic’s rate-limit page says limits “represent maximum allowed usage, not guaranteed minimums” and are enforced “at the organization level”, and that a request over a limit gets a 429 error “along with a retry-after header indicating how long to wait” (Anthropic, Rate limits). The same page describes a spend cap per usage tier. Both are things a design has to plan for.

Reserved capacity is buying throughput at a fixed cost. Amazon Bedrock’s documentation says you “can purchase Provisioned Throughput to provision a higher level of throughput for a model at a fixed cost”, that it is billed on a time basis, that the price depends on the model, the number of model units and the commitment level, and that “billing continues until you delete the Provisioned Throughput”. It lists a level with no commitment and levels with fixed terms, which “can’t” be deleted before the term ends and are discounted the longer the commitment. It also says that “if you customized a model, you must purchase Provisioned Throughput to be able to use it” (AWS, Increase model invocation capacity with Provisioned Throughput in Amazon Bedrock). That is one provider’s service, and its terms are not a statement about any other provider’s.

Self-hosted means the team runs the model. vLLM, an open library for this, describes itself as “a fast and easy-to-use library for LLM inference and serving” with “continuous batching of incoming requests” and an “OpenAI-compatible API server” (vLLM documentation). Choosing it removes a provider’s per-minute limits as the thing you hit, and replaces them with the capacity of the machines you provision. Who gets paged when it stops, who scales it up and who updates the model is now the team.

• Applied use. Choose on-demand when volume is low or unpredictable, reserved when volume is steady and high enough that a fixed cost beats paying per token, self-hosted when something a provider cannot give you (a model you must run yourself, or data that may not leave your environment) forces it.
• Signal. “Spiky”, “pilot”, “unknown volume” point to on-demand. “Steady high volume”, “we fine-tuned a model on the provider’s platform” point to reserved (for the Bedrock case the document says it is required). “Must run inside our own environment” or “open-weight model we control” points to self-hosted.
• Practical consequence. On-demand gives you limits and error handling to build around. Reserved gives a bill that does not shrink when traffic does. Self-hosted gives you on-call work, capacity planning and upgrades.
• Versus the distractor. Self-hosting is not “reserved but cheaper”. It trades a provider’s price for your own staff and machines, and it is the wrong answer when nothing in the requirements forces it and volume is low, because the fixed operating work exists whether or not traffic does.

What each place commits you to

Two situations make the commitments concrete. A team buys reserved capacity for a launch and traffic falls away after the launch: the fixed cost stays, because the bill depends on units and commitment, not on use. A team self-hosts to avoid per-minute limits and then has an overnight failure: there is no provider to call, and the fix is the team’s. The same shape of trade appears in the other places; you move the risk, you do not remove it.

Place	What you pay	What caps you	What you operate
Managed on-demand	Per token used, at the provider’s published price	The provider’s rate limits and spend cap	Retry and fallback handling in your client; nothing about the model itself
Reserved capacity	A fixed amount for the capacity, used or not	The capacity you bought (the number of units)	Choosing how many units, and deleting them when no longer needed
Self-hosted	Machines, plus the work of running them	The machines you provision	Serving software, scaling, failures, model updates, monitoring

The fourth column is the one that does not show up in a price comparison, which is why a design review asks for it separately.

The arithmetic of a fixed cost against a per-token price

Here is a worked example with invented figures. On-demand costs 5 cost units per million tokens. One reserved unit costs 800 cost units per day and serves up to 250 million tokens per day. The break-even is the volume where a fixed cost equals paying per token: 800 ÷ 5 = 160 million tokens per day. At 100 million, on-demand costs 500 and a unit costs 800, so on-demand is cheaper. At 200 million, on-demand costs 1,000 and one unit costs 800, so reserved is cheaper. At 300 million the picture changes again: units are bought whole, so you need two units, which cost 1,600, and on-demand costs 1,500, so on-demand is cheaper once more. Capacity comes in steps, so the cheapest place can change as volume rises.

A second way to see it is a phone plan. A pay-per-minute plan is cheapest for a person who rarely calls; a flat plan wins for a heavy caller; a heavy caller who needs a second line pays for a whole second plan.

Self-hosting adds a fixed operating cost that exists whatever the volume. If one server costs 300 per day and serves up to 80 million tokens per day, with a further 400 per day for the work of running it, then at 80 million it costs 700 against 400 on-demand. That fixed amount is a modelling choice, and in real deployments it is hard to price; the point is that leaving it out makes self-hosting look cheaper than it is.

• Applied use. Price each option at the expected volume and also at a high and a low estimate. Where the cheapest option changes inside that range, the volume forecast is the thing to challenge.
• Signal. A stem that gives a steady volume, a fixed price and a capacity per unit is asking for this arithmetic; “units cannot be split” is the cue that capacity comes in steps.
• Practical consequence. The forecast is an assumption, so the decision should state what volume would flip it, and the fixed-cost options need an owner who revisits it.
• Versus the distractor. The wrong answer divides the fixed cost by the per-token price and stops, forgetting that capacity above one unit is bought as a second whole unit.

Putting a workload and a place together

Batch work whose deadline is at least the window a provider states can go to that provider’s batch interface, which the sources above describe as discounted and on separate limits. Interactive work is priced across the three places at its volume. The two meet in the design: a single product often has both, and the useful move is to keep them apart so that one cannot starve the other.

Three situations you will meet, and what they mean for your design

1. A pilot assistant has twenty users and no usage history. Volume is unknown and small, and nobody has asked for the model to run inside your own environment.
2. A bank’s payment screening calls a model before each payment is approved. The call is made by the payment system, there is no person at a screen, and the payment waits for the answer.
3. A product team needs descriptions regenerated for a large catalogue, and the next release is soon but not tomorrow.

What you see	What shape and place	What to do
Unknown, small volume, interactive (situation 1)	Interactive; managed on-demand	Start on-demand, plan retries and watch the limits, and revisit when volume is known
A calling system waits, volume steady (situation 2)	Interactive, even though no person waits; price on-demand and reserved at the volume	A system waiting counts as waiting, so it cannot go to batch; choose by the arithmetic
Large, deadline measured against the stated window (situation 3)	Batch if the deadline is at least the window the provider states	Use the batch interface and keep it off the interactive account’s limits

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

How do you decide whether a workload is interactive or batch?

Ask who is waiting and when the result stops being useful. If a person or a calling system is waiting, it is interactive. If nobody is waiting and a result is needed by a deadline, it is batch, provided the deadline is at least as long as the window the batch interface states.

What do two providers state about batch interfaces?

OpenAI states a 24-hour turnaround, a 50% discount against synchronous calls, and a separate pool of limits. Anthropic states a 50% saving, most batches finishing in under an hour, and expiry if not complete within 24 hours, with expired requests not billed. Check the current page before relying on the figures.

Why did the chat users in the opening failure see errors?

The summary job and the chat shared one account’s per-minute limits, and the job used most of them, so interactive requests got 429 errors. A batch interface for the job would have kept it out of those limits.

What is managed on-demand inference, and what caps you on it?

A provider hosts and runs the model and you call it, paying for what you use. Its limits cap you: one provider’s page says they are maximum allowed usage rather than guaranteed minimums, and a request over a limit gets a 429 with a retry-after header.

What does reserved capacity cost, and when is it a good fit?

A fixed amount for a number of units, billed whether or not they are used. It fits steady volume high enough that the fixed cost beats paying per token, and in the Bedrock case it is required to use a customised model.

What does self-hosting commit a team to that on-demand does not?

Running the serving software and machines: scaling, failures, updates and monitoring. The provider’s per-minute limits stop being the cap and the capacity the team provisions takes their place.

What is a break-even volume, and why can it mislead?

The volume where a fixed cost equals paying per token. It misleads because capacity is bought in whole units, so past the first unit the cheapest place can change again, and because the volume is a forecast.

Why is a payment screen that no person watches still interactive?

A system is waiting for the answer: the payment cannot proceed until the call returns. The test is whether anything is waiting, not whether it is a human.

What commitment does Bedrock Provisioned Throughput carry?

It is billed on a time basis, the price depends on model, units and commitment level, longer commitments are discounted and cannot be deleted before the term ends, and billing continues until you delete it.

What does a design review ask that a price comparison misses?

What the team operates: on-demand needs retry and limit handling, reserved needs someone to size and delete units, and self-hosted needs on-call, scaling and updates. These do not show in a cost per million tokens.

Model answer

The question: “We are adding a model to a product. Where should the calls run?” An architect is asked this at a design review before a bill exists. A strong answer hits five beats, in this order:

1. Shape — who is waiting, and by when is the result needed: interactive or batch.
2. Place — for batch, a batch interface; for interactive, on-demand, reserved or self-hosted.
3. Numbers — price the options at the expected volume with stated assumptions, in whole units.
4. Commitment — name what each choice binds the team to operate or pay for.
5. Flip condition — say what volume or requirement would change the choice.

“I would start by asking who is waiting. If a person or a calling system is waiting, the workload is interactive; if nobody is and there is a deadline at least as long as the provider’s stated batch window, it is batch and goes to a batch interface, which is cheaper and uses a separate pool of limits so it cannot starve the interactive traffic. For interactive traffic I would price on-demand, reserved capacity and self-hosting at the expected volume, remembering that units are bought whole and that self-hosting carries a fixed cost for running it. I would name what each commits us to: retry handling and limits for on-demand, a bill that stays when traffic falls for reserved, and on-call, scaling and updates for self-hosted. I would start on-demand when volume is unknown, and I would say what volume or requirement, such as a model we must run in our own environment, would make me move.”

Missing a beat is a failure state. Without shape the answer picks a place before it knows what is waiting. Without place it does not choose. Without numbers it is a preference. Without commitment the cost looks smaller than it is. Without flip condition the choice cannot be revisited.

Why this matters

• Design reviews ask for it. A proposal to buy reserved capacity or to host a model is judged by the volume forecast behind it and by who will operate it.
• Money follows the shape. A discount and a separate pool of limits for batch work are offered on work submitted as batch, so misclassifying a job can cost money or capacity.
• Outages follow the mix. Putting a large background job on the account a chat product uses can turn a quiet evening into errors for users.
• Talking to customers. When a customer asks why their pilot was kept on pay-per-use, an architect who can say what volume would justify a fixed commitment gives a better answer than “it is cheaper”.

After this chapter

The project for this chapter is Place the Workloads (project.md). You will sort eight workloads into interactive or batch with a printed rule, price three places to run a model at four daily volumes, and finish by writing placement.tsv: each workload with its shape and its place.

Chapter 2 opens with the interactive workloads in your placement.tsv. It shows the shapes retrieval and agentic systems take, and asks which shape each of those workloads needs around the model call.

Hands-on Project — Place the Workloads

Objective

Sort eight workloads into interactive or batch with a printed rule, price three places to run a model at four daily volumes, and write a placement file, so that the choice of where a workload runs rests on a stated rule and stated numbers.

What this project needs: Python 3 and its standard library only. No task needs an account with a cloud or model provider, an API key, an internet connection or any payment. Every workload, price and capacity is printed below; the prices are invented teaching figures, not any provider’s prices. All three tasks are required, and Tasks 1 and 2 can be done in either order.

Your evidence file. Keep one plain-text file, chapter1-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print. Where a value is a list, write its items comma-separated with no spaces, in the order given.

The data and the rules for all tasks

Copy this into your Python file. Each workload is (id, what it does, someone_waiting, deadline_minutes, volume_m). someone_waiting is true when a person or a calling system cannot continue until the answer returns. deadline_minutes is how long after the request the result stays useful (0 when someone is waiting). volume_m is millions of tokens per day.

WORKLOADS = [
    ("w1", "A support chat reply",                              True,  0,     20),
    ("w2", "Nightly re-labelling of the day's tickets",         False, 2880,  120),
    ("w3", "An order lookup on a checkout page",                True,  0,     200),
    ("w4", "Summaries of an archive for tomorrow's review",     False, 1440,  500),
    ("w5", "Payment screening before a payment is approved",    False, 1,     500),
    ("w6", "An evaluation run for a decision due this afternoon", False, 600,  1200),
    ("w7", "A backfill of embeddings for an old archive",       False, 10080, 300),
    ("w8", "An agent step inside a user's open request",        True,  0,     20),
]
BATCH_WINDOW_MINUTES = 1440   # the outer window of the batch interface used here

The shape rule. For a workload, take the first that applies:

1. someone_waiting is true: the shape is interactive.
2. deadline_minutes is at least BATCH_WINDOW_MINUTES: the shape is batch.
3. Otherwise the shape is interactive.

The prices (all in cost units, invented):

• A, on-demand: 4 cost units per million tokens.
• B, batch: half of A, available only to a batch workload: 2 cost units per million tokens.
• C, reserved: 600 cost units per day for each unit; one unit serves up to 200 million tokens per day; units are bought whole, so the number of units is the volume divided by 200, rounded up.
• D, self-hosted: 250 cost units per day for each server plus a fixed 300 per day for running them, whatever the number of servers; one server serves up to 100 million tokens per day; servers are bought whole, so the number is the volume divided by 100, rounded up.

Task 1 — Sort the workloads by shape

First, a carry-in from the tool review in Building AI Agents. Its project reviewed an agent’s five tools, each written (name, can read, can write, can delete, needs approval), against three rules. Re-run them here:

NEEDED = {"lookup_order"}
TOOLS = [
    ("lookup_order", True, True,  False, False),
    ("update_order", True, True,  False, True),
    ("delete_order", True, False, True,  False),
    ("export_all",   True, False, False, False),
    ("refund",       True, True,  False, False),
]

The rules: excess_function when the tool’s name is not in NEEDED; excess_permission when the name is in NEEDED and the tool can write or delete; no_approval when the tool can write or delete and does not need approval. A tool can have several findings. Count the findings across all five tools.

Then write shape(workload) following the shape rule, and print the shape of all eight workloads and the number of each shape.

Add to your evidence file:

t1_carried_findings: <the total number of findings across the five tools>
t1_batch_ids: <ids whose shape is batch, in order, comma-separated, no spaces>
t1_interactive_count: <number of workloads whose shape is interactive>
t1_w5: <shape of w5>

Output: one Python file with a shape function, and four evidence lines.

Task 2 — Price the three places

Write a function for each of A, C and D returning the cost per day, in cost units, of a volume in millions of tokens per day, following the prices above. For each of these four volumes, find the cheapest of A, C and D: 20, 200, 500 and 1200. (B is not part of this comparison: it is for batch workloads only.) Then, for the batch workload w4, find its cost under B.

Add to your evidence file:

t2_cost_500: <cost at 500 million tokens per day, in the form a=N,c=N,d=N, whole numbers>
t2_cheapest: <the cheapest option letter, lower case, for volumes 20, 200, 500, 1200 in that order, comma-separated, no spaces>
t2_w4_batch_cost: <cost of w4 under B, whole number>

Output: three cost functions, and three evidence lines.

Task 3 — Write the placement file

This task needs the data, the shape rule and the cost functions from Task 2.

Write placement.tsv: one line per workload, in order, with three fields separated by single tab characters: the workload id, its shape, and its place. The place is:

• batch when the shape is batch;
• otherwise the cheapest of A, C and D at the workload’s own volume_m, written on-demand for A, reserved for C and self-hosted for D.

For the interactive workloads, no two of A, C and D tie for cheapest.

Add to your evidence file:

t3_rows: <number of lines in placement.tsv>
t3_reserved_ids: <ids whose place is reserved, in order, comma-separated, no spaces>
t3_selfhosted_count: <number of workloads whose place is self-hosted>

Output: a placement.tsv with one line per workload, and three evidence lines.

What carries forward

Keep placement.tsv and chapter1-evidence.txt. Chapter 2 opens with the workloads your file marks interactive: the ones where something is waiting while the model call runs. It asks what has to surround each of those calls, whether a retrieval step, a tool loop or neither, and what that shape locks in.