Chapter 5 — From Prototype to Production

Course: Taking AI to Production

Prerequisites: You can read and run a short Python script that uses the standard library, and you have seen a function call a service over HTTP and read its status code. You know what a token is and that a model service charges for the tokens it reads and writes. The idea of a chain of calls from an entry point to an outside call is restated in the notes. Every other term, including timeout, backoff and idempotent, is defined in the notes before it is used.

What the reader can do after this chapter:

• Choose a timeout for an outside call from a sample of its response times, and say what a call without one costs the caller.
• Decide for each error response whether to retry, compute the delays of a capped exponential backoff, and say when a retry makes an outage worse.
• Apply a budget to a stream of model requests: cap the output, fall back to a cheaper model, and refuse what no longer fits.

Core question this chapter answers: A demo of an AI feature works: what must be added around each outside call before real users, real failures and a real bill can be allowed near it?

Connections:

• Builds on: the idea of a chain of calls from an entry point to an outside call; each hop of that chain is a place where the protections in this chapter go.
• Leads into: Chapter 6, which makes the feature observable in production, so that each call, retry, refusal and cost can be seen and followed after it happens.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines. Every accepted value, or the rule that decides it, follows from what the task prints.

Taking AI to Production — Chapter 5: From Prototype to Production

Before you start

Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library; no account, API key or network is needed, because every response time, status and token count is printed in the project file and delays are added up instead of waited. Keep one question in mind while you read: what does this call do when the thing on the other end is slow, broken or expensive?

Summary

A prototype is shaped by the case where every call works, and production is shaped by the cases where one does not. Four protections go around each call that leaves your program. A timeout bounds how long you wait. A retry policy decides which failures are worth another attempt, how long to pause between attempts and when to stop, because a retry adds load to a service that may already be overloaded. A degraded path gives the user something smaller than the full answer instead of an error. A budget caps what each request and the whole stream of requests may cost, and decides what to do when the money is gone. Each is a few lines of code, and each is easiest to get right by replaying scripted failures rather than waiting for real ones.

Key concepts

Term	Definition
Prototype	A version built to show the feature working in the case where everything goes well; it need not survive failure.
Timeout	The longest a caller will wait for a reply before giving up on a call.
Deadline	A limit on the whole call, kept by the caller, covering every connection attempt, read and retry.
Retry	Sending the same request again after a failed attempt.
Backoff	A pause between retries that grows with each failed attempt, for example doubling.
Capped exponential backoff	A backoff that doubles each time up to a stated maximum pause, after which the pause stays at that maximum.
Jitter	Randomness added to a pause so that many callers do not all retry at the same instant.
Idempotent	Describes a request that can be sent more than once and have the same effect as sending it once.
Retryable error	A failure that may succeed if tried again, such as an overload or a rate limit that carries a wait time; as opposed to a failure that will repeat, such as a malformed request.
Degraded path	A planned smaller response, such as an earlier saved answer, a cheaper model or a queued job, used when the full path is unavailable.
Budget	A stated limit on cost, applied before a request is sent, so that spending is decided rather than discovered.
Output cap	A limit on how many tokens a model may write in one reply, which also bounds the cost of that reply.
Replay	Running new code against a scripted history of outcomes, such as a recorded failing request, and comparing the result with an expected value written in advance.

Explanations

The failure, before the explanation

A team demos a feature that summarises support tickets with a model. In the demo, every call returns in about a second. In the first week of use, the model service has a slow period, and each summary now takes a very long time to come back. The web workers that call it are each waiting on one reply, so the workers fill up, new tickets queue behind them, and pages that have nothing to do with summaries stop loading. Someone adds a quick fix: retry when a call fails. During the next overload, every worker retries, the service receives several times the traffic it had before, and it stays down longer. The same month, a loop in one script sends the same long document thousands of times, and the bill arrives as a surprise.

None of these is a model problem. They are what happens around a call when nothing limits how long it waits, how often it tries again, or how much it spends. A second picture: a shop that has one till, no queue limit and an open credit line. The shop works on a quiet day and fails exactly when it is busy.

Timeouts: decide how long to wait

Take a call to a summary service where most replies arrive in under half a second. With no timeout, a reply that takes forty seconds holds a worker for forty seconds. The Python documentation says the optional timeout parameter of urllib.request.urlopen “specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting will be used)”, and that it “actually only works for HTTP, HTTPS and FTP connections” (Python documentation, urllib.request). The socket documentation says a socket operation that runs past its timeout raises TimeoutError (Python documentation, socket), so a timeout turns an open-ended wait into an error the code can handle. Note what that timeout limits: each blocking operation, such as the connection attempt or one read, not the call as a whole. A reply that keeps arriving a little at a time can run past it in total. To limit the whole call, the caller also keeps its own deadline: note the start time, and stop reading or retrying once the deadline has passed.

Why waiting is costly is stated in the AWS Builders’ Library article on timeouts and retries: “When a client is waiting longer than usual for a request to complete, it also holds on to the resources it was using for that request for a longer time” (Amazon Web Services, Timeouts, retries, and backoff with jitter). The same article describes one way to pick a value: choose an acceptable rate of false timeouts, such as 0.1%, then look at the corresponding latency percentile of the downstream service; its example uses the 99.9th percentile (same source).

Here is that idea worked on a sample. Ten past response times in milliseconds: 80, 90, 95, 100, 110, 120, 130, 150, 900, 6000. Taking the value at the 80th percentile by the nearest-rank method (sort, then take the value at position ceil(0.8 * 10) = 8) gives 150. With a deadline of 150 on the whole call, two of ten calls (900 and 6000) would time out, and the caller gives up on each of them at about 150 instead of waiting 900 and 6000. A timeout at that value also cuts off two calls that would have succeeded, which is the price: the percentile you choose sets how many good calls you are willing to lose.

• Applied use: set a timeout on each call to an outside service, taken from a sample of that service’s own response times, back it with a deadline on the whole call, and handle the error either raises.
• Signal: a call with no timeout argument inherits whatever default the library or system has, which may be no limit at all; check the call, do not assume.
• Practical consequence: with a whole-call deadline, workers, connections and users are released at a time you chose, and a slow dependency stops being able to hold up unrelated work.
• Selection: choose a timeout from measured latency rather than a round number picked by feel; a timeout below the usual response time fails good calls, and one far above the worst reasonable time protects nothing.

Errors and retries: which to try again, and how

Two failed calls. One returned status 503, which means the service was temporarily unable to handle the request. The other returned status 400 with type invalid_request_error: the request itself was malformed. Sending the 400 again sends the same malformed request and gets the same answer; sending the 503 again may succeed. Anthropic’s API documentation describes 400 invalid_request_error as an issue with the format or content of the request, and 500 api_error as an unexpected internal error for which it says to retry with exponential backoff; it lists 529 overloaded_error as the API being temporarily overloaded and 504 timeout_error as a request that timed out while processing (Anthropic, Claude API errors). In Python, urllib’s default handler turns every HTTP error response into an HTTPError exception, whose code attribute holds the status (Python documentation, urllib.request and urllib.error, retrieved 2026-10-05), so the code can tell a 503 from a 400.

Rate limits have a wrinkle. A 429 response means a limit was hit. The Anthropic rate-limit documentation says a retry-after header gives the number of seconds to wait until the request can be retried and “earlier retries will fail”; it also says the 429 returned when a tier’s spend cap is reached has no retry-after header and that retrying, including the SDK’s automatic retries, fails until access resumes (Anthropic, Rate limits). Two 429s can therefore deserve different treatment: wait the stated time for one, and stop for the other. The same documentation says a spend limit that you set yourself returns a 400 when reached, so read the error type as well as the status: neither 400 succeeds on a quick retry, but they call for different fixes.

How long to pause between retries: with capped exponential backoff the pause doubles up to a maximum. For a base of 50 and a cap of 400 (milliseconds), the pauses after the first five failures are 50, 100, 200, 400 and 400. The AWS article notes that exponential backoff can lead to very long backoff times, so implementations typically cap it, and that jitter adds randomness to the backoff to spread retries around in time (same source). With no jitter, many callers that failed together retry together.

Retries also add load. The AWS article says “When failures are caused by overload, retries that increase load can make matters significantly worse”, and gives an example in which, if each layer retries independently, “the load on the database will increase 243x” (same source). The multiplication is easy to check: 243 is 3 to the fifth power, the count for five layers that each make three attempts. With three layers (a service calling a service calling a service) it is 3 * 3 * 3 = 27 attempts at the bottom; each layer multiplies. The article’s mitigation is to limit retries locally using a token bucket, so that retries are allowed while tokens last and then happen at a fixed rate. It also says APIs with side effects “aren’t safe to retry unless they provide idempotency”: a retried “send this email” can send two emails.

Anthropic’s documentation says its official SDK retries transient failures (connection errors, rate limits and 5xx server errors) with exponential backoff, twice by default, honouring retry-after when present, and that max_retries configures or disables this (Anthropic, Claude API errors, retrieved 2026-10-04). That is a reason to know what your client library already does before adding a second retry loop around it.

• Applied use: write down, per status, whether to retry; set a maximum number of attempts, a capped backoff and (for a real system) jitter; and keep one layer responsible for retrying.
• Signal: a retry loop with no maximum, no pause, or no distinction between a 400 and a 503.
• Practical consequence: a transient failure becomes a short delay, a permanent one fails fast, and an outage is not amplified by the callers.
• Selection: retry on a short list of statuses that can change on a second try; do not retry on a failure that will repeat, or on a request that is not idempotent without a way to make it safe. Retrying everything is the distractor, because it adds load while the service is already struggling.

Degrade instead of failing

Take the ticket summariser again. When the model service is down, the page can show an error, or it can show the most recent saved summary of that ticket marked as “may be out of date”, or the ticket’s first lines, or a note that the summary will appear when it is ready. A second example: when the large model is slow or the budget is nearly spent, answer from a smaller model instead of failing. These are degraded paths: planned, smaller responses, written before the failure.

This is design practice rather than something a vendor documents, so treat it as this chapter’s recommendation. A degraded path has to be chosen, built and tested like any other, and it needs a rule for when to use it: after a timeout, after retries are exhausted, or when a budget says no.

• Applied use: for each outside call, name what the user sees if it fails after retries.
• Signal: a feature whose only failure behaviour is an unhandled error page.
• Practical consequence: a failed dependency costs the user some quality, not the whole page.
• Selection: prefer a smaller, honest answer, labelled as such, to an error or to a stale answer shown as fresh; a degraded answer that hides its state is the distractor.

Cost control: decide before you send

Take a request with 600 input tokens and a model that may write 300 tokens in reply. Suppose the large model charges 3 cost units per input token and 15 per output token (invented numbers for this chapter). If the reply runs to all 300 tokens, the request costs 600 * 3 + 300 * 15 = 6300. Capping the reply at 200 tokens bounds it at 600 * 3 + 200 * 15 = 4800. The Messages API reference describes max_tokens as “the maximum number of tokens to generate before stopping”, and adds that the model may stop before reaching it (Anthropic, Messages API). So before sending you know only the worst case, 4800; the actual cost is known once the reply is complete and may be lower. A cap also bounds the answer’s length, so a cap set too low cuts good answers short.

Two protections stack. In your own code, a budget: before each request, compute its worst case (input plus the output cap) and send it only if spent + worst_case <= budget, then add its actual cost to spent once the reply is complete; otherwise use a degraded path (a cheaper model) or refuse. At the provider, a spend limit: the Anthropic documentation says you can set your own spend limit below your tier’s cap, that reaching it returns a 400, and that reaching the tier’s cap returns a 429 with no retry-after (Anthropic, Rate limits, retrieved 2026-10-04). Your own budget acts first and gives you a choice; the provider’s limit is the backstop that stops everything.

• Applied use: set an output cap; reserve each request’s worst case before sending it; keep a running total of actual costs; choose, in advance, what to do when a request does not fit.
• Signal: a loop or a batch job that sends requests with no total, no output cap and no stop.
• Practical consequence: the worst case for a request and for a day is a number you chose.
• Selection: a budget in the caller plus a provider limit as a backstop; relying only on the provider’s cap means the first sign of overspending is every request failing.

Replay: script the failure, then check the result

Take the ticket summariser again. Last week a call returned 503 twice and then a reply, and the page showed an error because the code gave up after one try. To fix it you change the retry policy. To know the fix works, you replay the failure: you write down that history (first attempt 503, second attempt 503, third attempt a good reply), feed it to the new code in place of the real service, and check the outcome. The old build fails the replay and the new build passes it. A second picture: a fire drill. Nobody waits for a real fire to learn whether the exits work, so the alarm is set off on purpose and the building is watched.

How to script one, as this chapter’s recommendation rather than a vendor rule. Replace the outside call with a stand-in that reads from a list of scripted outcomes, one per attempt, each a status and a response time. Run your retry, timeout and budget code against it. Then compare what happened with an expected value you wrote down before running: the number of attempts made, the total pause added up (not waited), the final status the user would see, and whether the degraded path was used. Because the script is fixed, the result repeats from run to run; if any randomness such as jitter is involved, seed it or check a range. A replay that prints results but compares nothing against an expected value cannot fail, so it proves nothing.

• Applied use: turn each past incident, and each failure the policy is meant to handle, into a scripted history with an expected outcome, and run them whenever the policy changes.
• Signal: a retry or fallback path that has not been run before the first real outage.
• Practical consequence: a change to the retry rules is shown to work or fail in seconds, without waiting for the service to break.
• Selection: prefer a scripted, repeatable replay with expected values to testing by hand against the live service, whose failures cannot be summoned on demand; a replay with no expected value is the distractor.

Three situations and what they mean for your design

1. A summariser’s workers all hang during a slow period at the model service.
2. After an outage a service starts receiving several times its usual traffic, and the callers’ logs show retries.
3. A nightly job sends every document in a folder to the large model.

What you see	What it means	What to do instead
Workers waiting with no error (1)	No timeout, so waiting is unbounded	Set a timeout from measured latency; handle the error; add a degraded path
Retry volume matches the outage (2)	Retries are adding load	Cap attempts, back off, add jitter, limit retries; retry only retryable statuses
Nightly job with no total (3)	The bill is whatever the folder holds	Add an output cap and a budget; fall back to a cheaper model; refuse at the limit

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

What does a call with no timeout cost the caller?

It holds the worker and its resources for as long as the reply takes, so a slow dependency can hold up work that has nothing to do with it.

How can a timeout be chosen?

From measured response times: pick an acceptable rate of false timeouts and take the matching percentile of the service’s latency. The percentile sets how many good calls you accept losing.

Why separate a 400 from a 503 when deciding to retry?

A 400 says the problem is on the caller’s side, such as a malformed request or a spend limit reached, so the same request fails again. A 503 means the service was temporarily unable to answer, so a later attempt may succeed.

What is capped exponential backoff?

A pause that doubles after each failed attempt up to a maximum, after which it stays at the maximum.

What does jitter do?

It adds randomness to the pause so that callers that failed together do not retry at the same instant.

How can retries make an outage worse?

They add load to a service that may already be overloaded; if each layer of a stack retries, the attempts multiply from layer to layer.

When is a retry unsafe?

When the request has side effects and is not idempotent, because sending it twice may do the thing twice.

Why can two 429 responses call for different handling?

One carries a retry-after header giving a wait time; the 429 for a spent spend cap has none and keeps failing until access resumes, so retrying it is wasted.

What is a degraded path?

A planned smaller response, such as a saved answer, a cheaper model or a queued job, used when the full path is unavailable.

What does an output cap do for cost?

It bounds how many tokens a reply may contain, so it bounds that reply’s output cost; set too low, it cuts answers short.

Model answer

The question: “A demo of your AI feature works. What do you add before real users use it?” A strong answer hits five beats, in this order:

1. Timeout — a bound on each outside call, taken from measured latency.
2. Retry — which errors, how many attempts, capped backoff and jitter, and idempotency.
3. Degrade — what the user sees when the call fails after retries.
4. Budget — an output cap, a running total and a rule for when it runs out.
5. Replay — tests that script the failures, so the paths are exercised before real ones occur.

“Each call that leaves the program gets a timeout chosen from how long that service really takes. Then a retry policy: only for errors that can change on a second try, a small number of attempts, a capped backoff with jitter, and only for requests that are safe to repeat. After the retries I decide what the user sees, such as a saved answer or a cheaper model, instead of an error. I cap the output, reserve each request’s worst case before sending it and keep a running total with a rule for when it is spent. And I test all of it by scripting failures, because real ones arrive at the worst time.”

Missing a beat is a failure state. Without a timeout, waiting is unbounded. Without retry rules, a blip becomes an outage or an outage becomes worse. Without a degrade path, a failed call reaches the user as an error. Without a budget, the bill is whatever the traffic makes it. Without replay, none of this is known to work.

Why this matters

• A launch. Real traffic can produce slow replies, rate limits and overloads that the demo did not show.
• A cost review. An output cap, a budget and a refusal rule are things a reviewer can read and a finance owner can set.
• An outage review. The questions asked afterwards are whether calls timed out, whether retries added load, and what users saw.
• Handing over. A retry table by status and a replay script are documents the next engineer can run.

After this chapter

The project for this chapter is Time It, Retry It, Budget It (project.md). You will choose a timeout from twenty response times, replay eight scripted request histories through a capped backoff retry policy, and run eight model requests through a budget with an output cap, a cheaper fallback and a refusal rule.

Chapter 6 makes the feature observable in production, so that each call, retry, refusal and cost can be seen and followed after it happens.

Hands-on Project — Time It, Retry It, Budget It

Objective

Choose a timeout from a sample of response times, replay eight scripted request histories through a capped backoff retry policy, and run eight model requests through a budget that caps output, falls back to a cheaper model and refuses what does not fit, so that you hold three repeatable protections and the numbers that show what each one does.

The data and the rules for all tasks

What this project needs: Python 3 and its standard library only (math). No task needs an account with a model provider, an API key, an internet connection or any payment. Nothing is run against a network and no code sleeps: every response time, status and token count is printed below, and delays are added up rather than waited. All three tasks are required and each can be done on its own, in any order.

Your evidence file. Keep one plain-text file, chapter5-evidence.txt. Each task says which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print.

The chain these protections go on. A chain of calls runs from an entry point to an outside call, such as handle_order>send_receipt>deliver>urlopen. This project treats the last call on a chain as the one to protect, and works on three things about it: how long to wait, whether to try again, and what each call may cost.

Task 1 — Choose the timeout

These are the response times, in milliseconds, of twenty past calls to one outside service. Each call did eventually reply.

LATENCIES = [120, 340, 95, 410, 150, 2800, 210, 180, 260, 90,
             5000, 300, 140, 220, 175, 380, 125, 240, 310, 160]

The rule for the timeout: sort the list from smallest to largest and take the value at position math.ceil(0.9 * n) counting from 1, where n is the number of calls. A call times out if its response time is greater than the timeout; the caller then gives up at the timeout and does not wait longer. A call that does not time out waits for its full response time. Work out the timeout, how many of the twenty calls time out, the total time the caller waits when this timeout is applied, and the total time it waits when no timeout is applied.

Add to your evidence file:

t1_timeout_ms: <the timeout>
t1_timeout_count: <number of calls that time out>
t1_wait_with_ms: <total wait with the timeout>
t1_wait_without_ms: <total wait with no timeout>

Output: a short script and four evidence lines.

Task 2 — Replay the retries

Eight requests were sent to a service. Each is listed with the status of each attempt in order. "429r" is a rate-limit response that carries a retry-after header; "429s" is a rate-limit response for a spent spending cap, which carries no retry-after header and keeps failing.

HISTORY = [
    ("r1", [200]),
    ("r2", [503, 200]),
    ("r3", [503, 503, 503, 503]),
    ("r4", [400]),
    ("r5", ["429r", "429r", 200]),
    ("r6", ["429s", "429s"]),
    ("r7", [500, 503, 503, 200]),
    ("r8", [503, 503, 503, 503, 200]),
]

The policy, applied to each request in turn:

• An attempt with status 200 ends the request as ok.
• Retry only after 500, 503 and "429r". Any other status (400, "429s") ends the request at once as stopped.
• At most 4 attempts in all. If the fourth attempt fails with a retryable status, the request ends as gave up. Statuses after the fourth attempt are never used.
• Before attempt k+1 (after the k-th failed attempt) wait min(300, 100 * 2**(k-1)) milliseconds. There is no wait after the last attempt, and no jitter in this task, so that every result is exact.

Add to your evidence file:

t2_attempts_total: <total attempts across the eight requests>
t2_ok_count: <number of requests that ended ok>
t2_delay_total_ms: <total of all waits, in milliseconds>
t2_gave_up_ids: <ids that gave up, id order, comma-separated, no spaces>
t2_stopped_ids: <ids that were stopped, id order, comma-separated, no spaces>

Output: a short script and five evidence lines.

Task 3 — Apply the budget

Cost is counted in abstract cost units. A model has a price per input token and per output token:

PRICE = {"large": (3, 15), "small": (1, 5)}   # (per input token, per output token)
MAX_OUT = 200        # output tokens above this are not generated
BUDGET = 7000        # total cost units allowed
REQUESTS = [("q1", 300, 120), ("q2", 150, 60), ("q3", 400, 350), ("q4", 200, 80),
            ("q5", 250, 150), ("q6", 100, 40), ("q7", 500, 90), ("q8", 120, 30)]

Each request is (id, input tokens, output tokens the model would write). The output actually written is min(output tokens, MAX_OUT). The cost on a model is input * price_in + output_written * price_out. Process the requests in order, keeping a running total spent. For each request: if spent plus its cost on the large model is at most BUDGET, serve it on the large model and add that cost. Otherwise, if spent plus its cost on the small model is at most BUDGET, serve it on the small model and add that cost. Otherwise refuse it and add nothing. Separately, work out how many cost units the MAX_OUT cap saves if every request were served on the large model: the total cost with each request’s full output, minus the total cost with output capped.

Add to your evidence file:

t3_large_ids: <ids served on large, id order, comma-separated, no spaces>
t3_small_ids: <ids served on small, id order, comma-separated, no spaces>
t3_refused_ids: <ids refused, id order, comma-separated, no spaces>
t3_spent: <final total spent>
t3_cap_saving: <cost units saved by the cap>

Output: a short script and five evidence lines.

What carries forward

Keep chapter5-evidence.txt and your three scripts. The output of Task 3, the budget applied to a stream of requests with its served and refused ids, is the one Chapter 6 opens with. That chapter makes an AI feature observable in production, where each call, retry and refusal leaves a record. A record is only useful if it says why a request was served or refused, so Chapter 6 starts by asking what your timeout, retry policy and budget should report about each request. Bring your three scripts and expect to change their constants, not their shape.