Chapter 3 — Scaling and Resilience for Inference

Course: Taking AI to Production

Prerequisites: You can read and run a short Python script. From earlier material you need three ideas, each restated in a sentence in the notes before it is used: a provider limits how fast an organisation may call a model and answers an excess request with an error (HTTP 429), the shape of a system sets how many model calls one request makes at worst, and a workload is interactive when someone is waiting and batch when nobody is. Concurrency, quota, backoff, jitter, queue, circuit breaker and degradation are defined in the notes before they are used.

What the reader can do after this chapter:

• Compute, from a request rate and a latency, how many requests are in flight, and compare the model-call rate to a quota.
• Produce a retry schedule with jittered exponential backoff, and trace a circuit breaker through a sequence of outcomes.
• Write a degradation table that says what a system does at each level of overload, and defend its order.

Core question this chapter answers: When demand for model calls exceeds the capacity you have, what keeps the system working, and what does it deliberately give up first?

Connections:

• Builds on: Chapter 2, whose architecture.tsv gave each request a worst-case number of model calls. This chapter opens with that file and turns the counts into load against a quota.
• Leads into: the data architecture chapter in Build and Evaluate RAG Applications, which opens with the degradation table this chapter’s project writes. Some of its modes serve cached or stale answers, and that chapter asks where such data lives and how stale it may be.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question, so you can check that your answer has the right shape before you submit it.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines and one tab-separated file. Every accepted value, or the rule that decides it, follows from what the task prints.

Taking AI to Production — Chapter 3: Scaling and Resilience for Inference

Before you start

Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library. You do not need an account with any cloud or model provider or an API key: the project does arithmetic on printed figures, runs a seeded random generator and traces a small state machine, and never calls a model. The figures are invented teaching numbers.

Keep one question in mind while you read: when there are more requests than capacity, what does the system do on purpose, and in what order?

Summary

Scaling inference starts with a sum: the number of requests in flight equals the arrival rate times the time each takes, and the rate of model calls is the request rate times the calls each request makes. That load meets a provider’s quota, and a request over the quota gets an error. The first responses are to retry with exponentially growing, randomised waits and to put a queue in front of work that can wait. When a dependency keeps failing, a circuit breaker stops sending it traffic so that it can recover and so that callers do not pile up. The last layer is a degradation plan chosen in advance: what the system gives up first, so that its core function keeps working when capacity runs out.

Key concepts

Term	Definition
Concurrency	The number of requests being handled at the same moment, that is, in flight.
Quota	The cap a provider or a platform puts on how much an organisation may use in a period, such as model calls or tokens per minute. A request over it is refused with an error (HTTP 429).
Backoff	Waiting before a retry, with the wait growing after each failed attempt.
Exponential backoff	Backoff in which the ceiling on the wait doubles with each attempt, usually up to a cap.
Jitter	Randomness added to a wait so that many clients do not retry at the same moment.
Retry amplification	Retries at several layers of a system multiplying the number of attempts that reach the bottom layer.
Queue-based load leveling	Placing a queue between the callers and a service so that bursts are stored and the service takes work at its own pace.
Circuit breaker	A proxy that counts recent failures and, past a threshold, fails calls immediately instead of sending them, then lets a few through to test recovery. Its states are closed, open and half-open.
Graceful degradation	Designing a component so that it still performs its core function, in a reduced form, when a dependency is unavailable.
Load shedding	Refusing or postponing some work on purpose so that the rest can be served.
Utilisation	Demand divided by capacity. Above 1.0, demand exceeds capacity.

Explanations

The failure, before the explanation

Here is a constructed example. A product launches an assistant. At launch, demand is higher than the team’s estimate, requests start to be refused with rate-limit errors, and the client library retries each refused request three times at once. The refusals continue, because the retries add to the demand that caused them. The chat users see spinners, and the team’s available dial is to raise the quota, which will take time. Nothing in the design said what should happen when capacity ran out, so the system chose for itself: it kept sending, and it failed for everyone together.

One provider’s documentation says that “unsuccessful requests contribute to your per-minute limit, so continuously resending a request won’t work” (OpenAI, Rate limits). An AWS engineering article says that “when failures are caused by overload, retries that increase load can make matters significantly worse” (AWS Builders’ Library, Timeouts, retries, and backoff with jitter). The rest of this chapter is the choices the team did not make.

Concurrency: how much is in flight, and how many calls that is

Two examples first. A kitchen takes 10 orders a minute and each order takes 3 minutes from start to finish: on average 30 orders are being cooked at any moment. An inference service receives 20 requests a second and each takes 3 seconds: on average 60 requests are in flight. This is Little’s law: the long-term average number in a system equals the average arrival rate times the average time each spends in it, written L = λW (Wikipedia, Little’s law). The article says the relationship “is not influenced by the arrival process distribution, the service distribution, the service order, or practically anything else”, and that the result “does not necessarily directly apply over finite time intervals” (same source). So it gives an average over the long run, and a short burst can exceed it.

Model calls are a second sum. If each of those 20 requests per second makes up to 4 model calls (the worst case set by the shape in Chapter 2), the call rate can be up to 80 calls per second, and that is the figure to compare with a quota. A second way to see it is a toll road: the number of cars on the road is arrival rate times time on the road, and the toll booths (the quota) care about cars per second, not cars on the road.

• Applied use. Compute requests in flight from rate and latency, and calls per second from rate and worst-case calls per request. The first sizes workers and connections; the second is compared with the quota.
• Signal. A stem that gives a rate and a time and asks “how many at once” is asking for L = λW; one that gives a rate and calls per request and a limit is asking for a quota comparison.
• Practical consequence. Slower requests mean more in flight at the same rate, so a change that adds latency, such as reranking or an agent loop, raises concurrency even if traffic does not change.
• Versus the distractor. The wrong answer sizes capacity from the average and ignores that the provider’s limit applies to short intervals: Anthropic’s page notes that a limit of 60 requests per minute “might be enforced as 1 request per second” (Anthropic, Rate limits).

Quotas and retries: backoff with jitter, and one place to retry

Anthropic’s rate-limit page says limits “represent maximum allowed usage, not guaranteed minimums”, that the API uses the token bucket algorithm so that “capacity is continuously replenished up to your maximum limit, rather than being reset at fixed intervals”, and that a request over a limit gets a 429 error “along with a retry-after header indicating how long to wait”. It also says a sharp increase in usage can trigger acceleration limits, so traffic should be ramped up gradually (Anthropic, Rate limits). When a provider sends a wait time, follow it: OpenAI’s page says to “follow Retry-After when it’s present and add a small random delay so multiple clients don’t retry at the same time”.

When there is no such hint, use exponential backoff with jitter. Here is one schedule. With a base wait of 1 second and a cap of 16 seconds, the ceiling on the wait before attempt n (counting from 0) is the smaller of 16 and 1 × 2 to the power n: 1, 2, 4, 8, 16, 16. Jitter then picks a random wait between 0 and that ceiling, the “full jitter” form. The AWS Architecture Blog gives it as sleep = random(0, min(cap, base * 2 ^ attempt)) and reports that without jitter the approach is “the clear loser”, since it “takes more work, but also takes more time”, and that “the return on implementation complexity of using jittered backoff is huge” (AWS Architecture Blog, Exponential Backoff and Jitter). A second way to see jitter is a crowd at a closed door: if everyone tries again exactly when the door might open, the door is mobbed again; if each person waits a different random time, arrivals are spread out.

Retries also need a limit on where they happen. The AWS article describes retry amplification: independent retries at each layer multiply the load on the layer below, and the example it gives of a five-layer stack increases the load on the database by 243 times, which is 3 to the power 5: 3 attempts per layer. (If each layer made 3 retries after its first attempt, that would be 4 attempts per layer and 4 to the power 5 = 1,024.) Its stated best practice is “to retry at a single point in the stack” (AWS Builders’ Library, Timeouts, retries, and backoff with jitter). The same arithmetic with two layers that each make up to 3 attempts gives up to 9 attempts at the bottom for one request.

• Applied use. Retry on throttling errors with backoff and jitter, honour a retry-after value when one is sent, cap the number of attempts, and retry at one layer only.
• Signal. “Thundering herd”, “many clients retry together” and “errors get worse after the first failures” point to missing jitter or missing limits on retries.
• Practical consequence. Retries trade latency for success, so the worst-case latency of a request now includes the waits, and a retry budget must be counted in the load figure.
• Versus the distractor. Immediate retries and fixed waits are the wrong answers, since synchronised clients return together. Raising concurrency to “work through” the errors is wrong when the cause is a quota, since the extra requests are refused and also count.

Queues: letting work that can wait wait

Two examples. A nightly job that labels tickets submits its requests into a queue, and a worker takes them at a rate that stays inside the quota while chat traffic is quiet. A burst of document uploads is stored in a queue and processed at the pace the model service can take. A second way to see it is a bank lobby with a ticket machine: the burst of arrivals is held in the queue, and the tellers work at their own pace.

Microsoft’s pattern page puts it this way: “use a queue that acts as a buffer between a task and the service that it invokes”, which “smooths intermittent heavy loads”. It says the queue decouples the tasks from the service “so that the service can handle the messages at its own pace”, and it also lists limits: “if your average producer rate exceeds the consumer rate, the queue continues to grow and latency increases”, and the pattern “might not be suitable” when “the caller requires a low-latency, synchronous response” (Microsoft, Queue-Based Load Leveling pattern).

• Applied use. Put a queue in front of work whose callers are not waiting, such as the batch workloads of the earlier chapters, and release it at a rate inside the quota.
• Signal. “Bursty”, “nobody is waiting” and “protect the model service from spikes” point to a queue; “the user is waiting” points away from it.
• Practical consequence. A queue converts overload into delay, so it needs a depth to monitor and a decision for when the delay is too long, and the work in it must be safe to process twice, since the page notes that most queue services deliver at least once.
• Versus the distractor. A queue is the wrong answer for an interactive request that needs an answer now, and an unbounded queue is not a fix when the average arrival rate exceeds the average service rate.

Circuit breakers and failover: stop sending to what is failing

Two examples. A model provider returns errors for many calls in a row; the application stops calling it for a short spell and answers with a simplified response instead. A shard of a data store is down; calls to that shard fail at once while calls to other shards continue. The household version is the electrical breaker that trips when the current is too high, so that the wiring is not damaged, and is switched back on only after the fault is cleared.

Microsoft describes the circuit breaker as a proxy that monitors recent failures and either lets an operation proceed or returns an exception immediately, implemented as a state machine. In the closed state requests go through and failures are counted; when failures pass a threshold within a period, the breaker moves to open, where “the request from the application fails immediately”, and starts a time-out. When the time-out ends it moves to half-open, where “a limited number of requests from the application are allowed to pass through”; if they succeed it returns to closed and the failure count is reset, and if any fails it returns to open. The page says the pattern serves “a different purpose than the Retry pattern” and may be combined with it, with retry logic that stops when the breaker says the fault is not transient. On what an application does when the breaker is open, it lists “temporarily degrade its functionality, invoke an alternative operation to try to perform the same task or obtain the same data, or report the exception to the user” (Microsoft, Circuit Breaker pattern).

That list is where failover sits: the alternative operation can be a second model, a cached answer or a simpler shape. Whatever the alternative is, its own capacity and quota must be known, because sending the failed traffic to it can overload it.

• Applied use. Wrap calls to a model provider in a breaker, set the threshold and the open time to match how long the provider’s faults usually last, and decide in advance what the open state returns.
• Signal. “Cascading failure”, “callers pile up waiting for time-outs” and “protect a recovering service” point to a breaker.
• Practical consequence. While a breaker is open, requests fail fast and cheaply, but some succeed less than they could have, and the open state must be observable so that people know it has tripped.
• Versus the distractor. Retrying harder is the wrong answer to a dependency that is down, since the page says a circuit breaker prevents an application from performing an operation that is likely to fail. A breaker is the wrong answer when the platform already handles failure recovery or when waiting for it to reset would cause unacceptable delay; the page lists both as cases where it may not suit.

Degradation: deciding in advance what to give up

Imagine the assistant with four things it can do: answer from retrieval, run an agent loop for investigations, refresh a nightly summary and show a status page. When capacity is short, something has to give. An unplanned system gives up whatever happens to time out. A planned one gives up in a chosen order: pause the nightly summary first, serve simpler or cached answers next, and refuse new work with a clear message last.

AWS’s reliability guidance says components “should continue to perform their core function even if dependencies become unavailable”, possibly “serving slightly stale data, alternate data, or even no data”, and lists as an anti-pattern “not identifying the core business functionality needed”. It says that deciding what to drop “is as much a business discussion as a technical one”, and that the fallback pathways “need to be tested and should be significantly simpler than the primary pathway” (AWS, REL05-BP01 Implement graceful degradation). The same page adds that, generally, fallback strategies should be avoided, so a degradation mode is something to choose deliberately, test, observe and keep simpler than the primary path.

• Applied use. Write the ladder down: the modes, the signal that moves the system from one to the next (utilisation is a simple signal), and what each mode serves.
• Signal. “Peak events”, “quota exhausted”, “what happens when the provider is slow” point to a degradation plan.
• Practical consequence. Each mode needs a test, a message to the user and a metric, and the low-priority work it pauses needs somewhere to wait.
• Versus the distractor. “Scale up” alone is the wrong answer when the cap is a quota that cannot be raised quickly, and “fail everything equally” is the wrong answer when some functions matter more.

Three situations you will meet, and what they mean for your design

1. Peak traffic reaches 20 percent more model calls per second than the quota, and a nightly job is also running.
2. The model provider returns errors on most calls for a while, and the support assistant is the front door of the business.
3. A launch doubles traffic, and the team cannot get a higher quota before it starts.

What you see	What to do	Why
Over quota by a margin; batch work running (situation 1)	Defer the batch work through a queue first, keep interactive traffic on the quota	Work that can wait should absorb the overload before users do
Provider failing; the assistant is core (situation 2)	Trip a circuit breaker and serve a simplified or cached answer, with a clear message	It stops piling callers onto a failing service and keeps the core function
Demand twice the quota, no quota increase (situation 3)	Walk down a written degradation ladder, ending in a clear refusal	The order is chosen and tested instead of decided by time-outs

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

What does Little's law say, and what does it give you?

The long-term average number in a system equals the average arrival rate times the average time in the system, L = λW. For inference it gives the average number of requests in flight from a rate and a latency. It is a long-run average, so a short burst can exceed it.

How do you compare a design with a model-call quota?

Multiply the request rate by the worst-case model calls per request to get calls per second, and compare that with the quota. The quota cares about calls per second, not requests in flight.

What does a provider's 429 with a retry-after header tell a client?

That a limit was exceeded and how long to wait. The client should wait at least that long and add a small random delay, because earlier retries fail and because many clients waiting the same time retry together.

What is full jitter, and why use it?

A wait chosen at random between 0 and the smaller of the cap and the base times 2 to the attempt number. It spreads retries in time, and the AWS Architecture Blog reports it using the least work among the variants it compared.

What is retry amplification, and what is the guidance?

Retries at several layers multiplying the attempts that reach the bottom layer; a five-layer stack with 3 attempts per layer is described as raising database load 243 times (3 to the power 5). The stated best practice is to retry at a single point in the stack.

When does a queue help, and when does it not?

It helps with bursty work whose callers are not waiting, since the service takes messages at its own pace. It does not suit a caller that needs a low-latency synchronous response, and a queue grows without bound if the average arrival rate exceeds the service rate.

What are the three states of a circuit breaker?

Closed, where calls pass and failures are counted; open, where calls fail immediately; and half-open, where a limited number of calls pass to test recovery. Success in half-open closes it; a failure reopens it.

What can an application do while a breaker is open?

Degrade its functionality, invoke an alternative operation for the same task, or report the failure to the user. The alternative’s own capacity has to be known, since the diverted traffic can overload it.

What does graceful degradation require you to decide first?

The core function, since that is what must keep working. Then which functions are dropped, in what order, and what the system serves in each reduced mode. AWS calls this as much a business discussion as a technical one.

Why must a degradation mode be tested?

A path that runs only in a bad period is otherwise the least exercised code in the system. AWS says such pathways need to be tested and should be significantly simpler than the primary pathway.

Model answer

The question: “Demand for our model calls will exceed our quota at peak. How do you keep the system working?” This is the question a reliability review asks before a launch. A strong answer hits five beats, in this order:

1. Size — compute requests in flight and model calls per second, and compare them with the quota.
2. Retry safely — jittered exponential backoff, a cap, a retry at one layer only, and the provider’s retry-after value honoured.
3. Level — put work that can wait behind a queue, released inside the quota.
4. Contain — a circuit breaker around the dependency, with a stated alternative when it is open.
5. Degrade — a written order of what is paused, simplified or refused, with a signal for each step.

“First I would size it: requests in flight from rate times latency, and calls per second from rate times worst-case calls per request, compared with the quota. I would retry only on throttling and server errors, with exponential backoff and full jitter, honouring any retry-after header, capped, and at one layer so that retries do not multiply. Work that can wait, such as the batch jobs, goes behind a queue and is released inside the quota. I would put a circuit breaker around the provider with an alternative when it is open, a simpler or cached answer, and I would know the alternative’s own capacity. Finally I would write the degradation ladder in advance: pause the batch work first, simplify next, refuse new work last with a clear message, with utilisation as the signal, and I would test each mode before launch.”

Missing a beat is a failure state. Without size the plan has no number to defend. Without retry safely the first errors make things worse. Without level work that can wait competes with users. Without contain a failing dependency drags everything with it. Without degrade the system chooses what to lose by accident.

Why this matters

• Launches fail here. The first day of real traffic is when a quota, a retry storm or a failing dependency is found, and a written plan is what turns an outage into a reduced service.
• Costs follow retries and loops. Retries, agent loops and reranking each add calls, and the load figure has to include them.
• Reviews ask for the order. A design review wants to see what is given up first, and why, because that is a business decision.
• Talking to customers. When a customer asks what happens at their busiest period, an architect who can say what pauses, what is simplified and what message users see gives a better answer than “it scales”.

After this chapter

The project for this chapter is Size the Load and Plan the Fall-back (project.md). The project gives you the content of Chapter 2’s architecture.tsv and you turn it into load figures against a quota, produce a seeded retry schedule and trace a circuit breaker, and finish by writing degradation.tsv: each overload situation with its utilisation and its mode.

The data architecture chapter in Build and Evaluate RAG Applications opens with your degradation.tsv. One of its modes serves cached or reduced answers, and that chapter asks where such data is kept, how old it may be, and how an answer says so.

Hands-on Project — Size the Load and Plan the Fall-back

Objective

Turn a table of requests and their worst-case model calls into load figures against a quota, produce a seeded retry schedule, trace a circuit breaker through a sequence of outcomes, and write a degradation table, so that what a system does when capacity runs out is a stated rule with stated numbers.

What this project needs: Python 3 and its standard library only (random is used in Task 2). No task needs an account with a cloud or model provider, an API key, an internet connection or any payment. Everything the tasks use is printed below, and the figures are invented teaching numbers. All three tasks are required, and each can be done on its own, in any order.

Your evidence file. Keep one plain-text file, chapter3-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print. Where a value is a list, write its items comma-separated with no spaces, in the order given.

The data and the rules for all tasks

The carry-in. This is the content of Chapter 2’s architecture.tsv with each request’s worst-case latency and a peak rate added. Each row is (id, shape, worst-case model calls, worst-case latency in seconds, peak requests per second):

LOAD = [
    ("r1", "single-call",   1, 2.0,  5),
    ("r2", "retrieval",     1, 2.3,  4),
    ("r3", "hybrid-rerank", 1, 2.8,  3),
    ("r4", "workflow",      3, 6.0,  2),
    ("r5", "agent",         8, 16.0, 0.5),
    ("r6", "single-call",   1, 2.0,  6),
    ("r7", "hybrid-rerank", 1, 2.8,  2),
    ("r8", "agent",         8, 16.0, 0.5),
]
QUOTA_CALLS_PER_SECOND = 30

Little’s law, used in Task 1: the long-term average number of requests in flight equals the average arrival rate times the average time each takes, so for one row it is the peak rate times the worst-case latency. Treat the peak rate and the worst-case latency as the averages.

Task 1 — Size the load

For each row, compute its model calls per second (peak rate times worst-case calls) and its requests in flight (peak rate times worst-case latency). Add each across the eight rows.

Add to your evidence file:

t1_calls_per_second: <total model calls per second across the eight rows>
t1_in_flight: <total requests in flight across the eight rows, one decimal place>
t1_over_quota_by: <total calls per second minus QUOTA_CALLS_PER_SECOND, as a whole number, 0 if not over>
t1_agent_calls_per_second: <model calls per second from the rows whose shape is agent>

Output: a short script, and four evidence lines.

Task 2 — Retry schedule and circuit breaker

Part A, a jittered retry schedule. Compute the wait before each of 6 attempts, numbered 0 to 5, with full jitter: the wait for attempt n is a random number between 0 and min(CAP, BASE * 2 ** n), with BASE = 0.5 and CAP = 8.0. Use Python’s generator seeded once with 42, and draw one number per attempt, in order of attempt:

import random
rng = random.Random(42)
wait = rng.uniform(0, min(CAP, BASE * 2 ** n))   # for n = 0, 1, 2, 3, 4, 5, in that order

Part B, a circuit breaker. Trace this simplified breaker (it counts calls instead of using a timer).

• It starts closed with a consecutive-failure count of 0.
• In closed, the call is sent. If its outcome is F, add 1 to the count; if the count reaches 3, the state becomes open with 4 rejections to go, and the count resets to 0. If the outcome is S, the count resets to 0.
• In open, the call is rejected without being sent, and the rejections to go drop by 1. When they reach 0, the state becomes half-open.
• In half-open, the call is sent. If its outcome is S, the state becomes closed with a count of 0. If it is F, the state becomes open with 4 rejections to go.

Each call in the list is the outcome it would have if it were sent. Run this list, one call after another:

OUTCOMES = "SSFFFSSSSFSSSSSFFF"

Add to your evidence file:

t2_waits: <the six waits in attempt order, each to three decimal places, comma-separated, no spaces>
t2_wait_total: <the sum of the six waits, two decimal places>
t2_rejected: <number of calls rejected while open>
t2_sent: <number of calls sent>
t2_final_state: <closed, open or half-open after the last call>

Output: a short script, and five evidence lines.

Task 3 — Write the degradation table

Utilisation is demand divided by capacity. Use whole-number arithmetic to avoid rounding: for a situation with demand and quota in model calls per second, take the first that applies:

1. demand * 100 <= quota * 100: the mode is full.
2. demand * 100 <= quota * 120: the mode is defer-batch.
3. demand * 100 <= quota * 150: the mode is simplify.
4. Otherwise the mode is refuse-new.

These thresholds are teaching numbers, not a recommendation for a real system. The situations, each (id, demand, quota):

SITUATIONS = [
    ("s1", 20, 30),
    ("s2", 30, 30),
    ("s3", 34, 30),
    ("s4", 36, 30),
    ("s5", 45, 30),
    ("s6", 60, 30),
]

Write degradation.tsv: one line per situation, in order, with three fields separated by single tab characters: the situation id, the utilisation as demand divided by quota rounded to two decimal places, and the mode.

Add to your evidence file:

t3_rows: <number of lines in degradation.tsv>
t3_modes: <the six modes in order, comma-separated, no spaces>
t3_refuse_count: <number of situations whose mode is refuse-new>

Output: a degradation.tsv with one line per situation, and three evidence lines.

What carries forward

Keep degradation.tsv and chapter3-evidence.txt. The data architecture chapter in Build and Evaluate RAG Applications opens with your degradation.tsv: the mode defer-batch postpones work, and the mode simplify serves reduced or cached answers. It asks where that data is kept, how old it may be before it is wrong, and how an answer says so.