Chapter 2 — Reference Architectures for Retrieval and Agentic Systems

Course: Taking AI to Production

Prerequisites: You can read and run a short Python script. From earlier material you need three ideas, each restated in a sentence in the notes before it is used: retrieval looks facts up in your documents and puts them in the prompt, a tool call is a request by the model for an outside system to do something, and a workload can be interactive (someone is waiting) or batch (nobody is). Single call, hybrid retrieval, reranking, workflow and agent are defined in the notes before they are used.

What the reader can do after this chapter:

• Choose among a single model call, a retrieval pipeline, a retrieval pipeline with hybrid search and reranking, a fixed workflow and an agent loop from a requirement’s stated properties.
• Compute the worst-case number of model calls, the worst-case latency and a chain success probability for a shape from stated figures.
• Name the trade-off each shape locks in (calls, latency, predictability, a component to maintain) and say what requirement would justify moving to the next shape.

Core question this chapter answers: Given a requirement for a system that answers from documents or acts through tools, how do you choose the simplest architecture that meets it, and what does that choice lock in?

Connections:

• Builds on: Chapter 1, which sorted workloads by shape and place. This chapter opens with the interactive workloads in the placement.tsv that chapter’s project wrote, and asks what has to surround each model call.
• Leads into: Chapter 3, which scales and protects inference. It opens with the architecture.tsv this chapter’s project writes, whose model-call counts are the load Chapter 3 sizes capacity for.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question, so you can check that your answer has the right shape before you submit it.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines and one tab-separated file. Every accepted value, or the rule that decides it, follows from what the task prints.

Taking AI to Production — Chapter 2: Reference Architectures for Retrieval and Agentic Systems

Before you start

Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library. You do not need an account with any cloud or model provider or an API key: the project picks an architecture from a printed rule and does arithmetic on printed figures, and never calls a model. The figures are invented teaching numbers.

Keep one question in mind while you read: what is the simplest arrangement of model calls, lookups and tools that meets this requirement, and what has it locked in?

Summary

A system built around a model takes one of a few shapes, and each shape adds something the previous one did not have. A single call asks the model once. A retrieval pipeline first looks facts up in your documents and puts them in the prompt, and it can be strengthened by combining search by meaning with search by exact words and by reranking the candidates. A workflow chains calls and tools along a path written in advance, and an agent lets the model decide its own next step in a loop. Each addition buys capability and costs something: more calls, more latency, a component to keep current, or less predictable behaviour and a risk that errors compound. The sound habit, which one major vendor states directly, is to find the simplest solution that works and add complexity only when a requirement needs it.

Key concepts

Term	Definition
Single call	One request to a model with the prompt you wrote, and one answer back, with no lookup and no tool.
Retrieval pipeline	A system that breaks documents into chunks, converts each chunk into a vector (an embedding) that captures its meaning, stores the vectors, and at request time finds the chunks closest to the question and puts them in the prompt.
Embedding search	Searching by meaning: a question and a chunk are close when their vectors are close, even if they share no words.
Lexical search (BM25)	Searching by exact words: BM25 is a ranking function that scores chunks by matching words and phrases in the question.
Hybrid retrieval	Running both embedding search and lexical search and combining their results.
Reranking	Taking the top candidates from the first retrieval, passing each with the question through a reranking model that scores them again, and keeping the best few.
Workflow	A system where models and tools are orchestrated through predefined code paths: the sequence of steps is written in advance.
Agent	A system where the model dynamically directs its own process and tool use, deciding the next step in a loop.
Stopping condition	A rule that ends an agent loop, such as a maximum number of iterations.
Compounding error	A mistake in an early step that is carried into later steps, so more steps give more chances to go wrong.
Augmented model	A model together with retrieval, tools and memory that it can use; the basic building block of the shapes above.

Explanations

The failure, before the explanation

Here is a constructed example. A team is asked for an assistant that answers questions from the company’s policy handbook. A senior engineer, excited by agents, builds one: the model is given a search tool and a calculator and allowed to loop until it decides it is done. In the demo it works. In use, a simple question like “how many days of leave do I get?” takes several model calls, because the model searches, reads, searches again and then checks its own answer. Latency is several times what the team expected, the bill per question rises with the number of loops, and on one question the loop wanders into the wrong chapter and answers confidently from it.

The requirement was “answer from the handbook”. It had known steps: find the relevant passage and answer. It did not need a model that chooses its steps. The team bought flexibility it had no use for and paid for it in calls, latency and a failure mode. This chapter is about choosing the shape the requirement needs.

The simplest shape: a single call

Two examples first. A user pastes a paragraph and asks for a friendlier tone: the model needs nothing it was not given. A ticket is to be labelled billing, technical or other: the model needs only the ticket and the three labels. A second way to see it is asking an expert a question from memory: if the question is about their craft and needs no reference, you get an answer in one exchange.

Anthropic’s guidance on agents states the habit: “When building applications with LLMs, we recommend finding the simplest solution possible, and only increasing complexity when needed.” It adds that “agentic systems often trade latency and cost for better task performance” (Anthropic, Building effective agents).

• Applied use. If the task needs only what is in the request, one call with a good prompt is the architecture.
• Signal. Wording such as “rewrite”, “classify”, “summarise this text” and no mention of documents or systems points to a single call.
• Practical consequence. One call is one unit of latency and cost, with nothing to maintain beyond the prompt.
• Versus the distractor. A retrieval pipeline or an agent is the wrong answer when nothing in the requirement needs facts the request does not carry or actions outside the model.

Retrieval: looking facts up before answering

Take the handbook question. A retrieval pipeline does the lookup that the single call could not. The original paper on retrieval-augmented generation describes the idea as pairing “a pre-trained seq2seq model” with “a dense vector index of Wikipedia, accessed with a pre-trained neural retriever”, and states that “providing provenance for their decisions and updating their world knowledge remain open research problems” for models on their own (Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks). The standard pipeline, as one vendor describes it, breaks the knowledge base “into smaller chunks of text, usually no more than a few hundred tokens”, converts the chunks into vector embeddings and stores them “in a vector database that allows for searching by semantic similarity” (Anthropic, Contextual retrieval). That is an open book with an index: the model reads the page the index points to.

A second way to picture the stronger forms: finding a recipe in a cookbook. Searching by meaning is asking the librarian for “something with chickpeas that’s quick”. Searching by exact words is looking up “shakshuka” in the index. Each finds what the other can miss.

That is the reason for hybrid retrieval. Embedding search captures meaning; BM25 is “a ranking function that uses lexical matching to find precise word or phrase matches”. A requirement with error codes, part numbers or product names, such as “find the fix for error E-4471”, is a case where exact words matter and embeddings can blur the identifier. Reranking adds a step: do “initial retrieval to get the top potentially relevant chunks”, pass them “along with the user’s query, through the reranking model”, and keep the top few.

The same article reports results on its own test datasets, which is worth stating as such: contextual embeddings “reduced the top-20-chunk retrieval failure rate by 35%”, combining them with BM25 reduced it “by 49%”, and adding reranking reduced it “by 67%” (Anthropic, Contextual retrieval). These are one vendor’s measurements on its datasets, not a promise for yours; the lesson for design is that each extra stage was reported to help and each is a component to run.

• Applied use. Use retrieval when the answer depends on documents the model was not trained on or that change; move to hybrid retrieval with reranking when queries contain exact identifiers or the first retrieval returns the right chunk too far down the list.
• Signal. “Answer from our documents”, “cite the source” point to retrieval; “error codes”, “part numbers”, “exact product names” point to the hybrid form.
• Practical consequence. The index must be built and kept current when documents change. Hybrid search adds a second index or ranking to maintain, and reranking adds a model call and some latency on every request. The retrieval step is the first thing to measure when an answer is wrong.
• Versus the distractor. A bare vector search is the wrong answer when exact identifiers matter, and the heavy hybrid-plus-rerank pipeline is the wrong answer for a small handbook where plain retrieval already returns the right chunk.

Workflow: a path written in advance

Two examples. A refund request is handled in three fixed steps: look up the order, check the refund rule, draft the reply. A marketing team generates copy and then translates it. A second way to see it is an assembly line: each station does one job and passes the part on, and the sequence does not change from one part to the next.

Anthropic describes this as workflows, “systems where LLMs and tools are orchestrated through predefined code paths”. Its patterns include prompt chaining (sequential steps, each processing the last output), routing (classifying an input and directing it to a specialised follow-up), parallelization, orchestrator- workers and evaluator-optimizer (Anthropic, Building effective agents). This chapter teaches the first two and treats the others as names to recognise. The source’s examples are “generating marketing copy, then translating it into a different language” for chaining and “directing different types of customer service queries (general questions, refund requests, technical support) into different downstream processes, prompts, and tools” for routing.

• Applied use. Use a workflow when the steps can be written down in advance, even if one of them is a retrieval step.
• Signal. “Then”, “after that”, “the same three steps each time” point to a workflow.
• Practical consequence. The number of model calls is fixed by the design, so cost and latency can be stated beforehand and each step can be tested on its own. The source says workflows “offer predictability and consistency for well-defined tasks”.
• Versus the distractor. An agent is the wrong answer when the steps are known, because it spends calls deciding what the code could have decided.

Agent: the model chooses the next step

Two examples. A coding task where the agent reads logs, forms a hypothesis, tries a fix and re-runs the tests, and what it does next depends on what the last run showed. A reconciliation task where which checks are needed depends on what the first checks find. A second way to see it is the difference between a recipe and a contractor: a recipe lists the steps; a contractor is told the outcome and decides the steps.

Anthropic defines agents as “systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks”. It says agents suit “open-ended problems where it’s difficult or impossible to predict the required number of steps, and where you can’t hardcode a fixed path”. It also says that “the autonomous nature of agents means higher costs, and the potential for compounding errors”, that the model needs “ground truth” from the environment at each step, such as tool results, and that it is common to “include stopping conditions (such as a maximum number of iterations) to maintain control” (Anthropic, Building effective agents).

Compounding can be seen with simple arithmetic. If each step is right with probability 0.9 and steps are independent, a five-step chain is right with probability 0.9 × 0.9 × 0.9 × 0.9 × 0.9, about 0.59. The independence is an assumption of the example and real steps are rarely independent, but the direction holds: more steps give more chances for an error to enter and carry forward.

• Applied use. Use an agent when the path cannot be written in advance and you can trust its decisions in the environment it runs in, with a maximum number of iterations.
• Signal. “Depends on what it finds”, “unknown number of steps”, “investigate” point to an agent.
• Practical consequence. The worst case is set by the stopping condition, not by the design: with a cap of eight iterations, a request can use up to eight model calls, and the latency and cost of a request vary. Its tools should be reviewed as in the tool review in Building AI Agents.
• Versus the distractor. A workflow is the wrong answer when the number of steps cannot be known, and an agent is the wrong answer when it can.

What each shape locks in

Shape	Adds	Worst case per request	Locks in
Single call	Nothing	One model call	A prompt to maintain
Retrieval	An index and a lookup	One lookup and one model call	An index kept current when documents change
Hybrid with reranking	A second search and a reranking step	One lookup, one rerank and one model call	Two retrieval components and a reranker to run
Workflow	Fixed steps and tools	The designed number of calls	Predictable cost, and a path to change in code
Agent	A loop and the model’s choices	The stopping condition’s number of calls	Variable cost and latency, and compounding error

Shapes also compose: a workflow can contain a retrieval step, and an agent can be given retrieval as one of its tools. Choosing a shape is choosing the outermost structure.

Three situations you will meet, and what they mean for your design

1. A product team wants answers from a handbook of a few hundred pages, with the source cited.
2. A support team wants to find fixes by error code in a large runbook collection, where users paste the exact code.
3. An operations team wants a system to investigate failed deployments, where the checks needed depend on what each earlier check shows.

What you see	Shape	What to do
Documents, cite the source (situation 1)	Retrieval	Build and measure the retrieval first; add stages only if it fails
Exact identifiers in queries (situation 2)	Hybrid retrieval with reranking	Combine lexical and embedding search and rerank, accepting the extra components
Path depends on what it finds (situation 3)	Agent with a stopping condition	Set the cap, give it only the tools it needs, and review them

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

What habit does one vendor recommend when choosing a shape?

Find the simplest solution possible and increase complexity only when needed. It notes that agentic systems often trade latency and cost for better task performance, so the trade should be justified.

What does a retrieval pipeline do before the model answers?

It breaks documents into chunks, converts them to embeddings, stores them, and at request time finds the chunks closest to the question and puts them in the prompt, so the model answers from them.

Why combine embedding search with BM25?

Embeddings capture meaning and BM25 matches exact words and phrases. A query with an identifier such as an error code is a case where exact matching matters and meaning-based search can miss it, so the two cover each other’s gaps.

What does reranking add, and what does it cost?

A second scoring of the top candidates from the first retrieval, with the question, so the best few go to the model. It costs an extra model call and some latency on each request, and one more component to run.

What results did one vendor report for these stages, and how far do they carry?

Reductions in the top-20-chunk retrieval failure rate of 35%, 49% and 67% for contextual embeddings, adding BM25, and adding reranking. They are that vendor’s measurements on its own datasets, so measure on yours.

What is a workflow, and what does it make predictable?

A system where models and tools are orchestrated through predefined code paths. The number of model calls is fixed by the design, so cost and latency can be stated in advance and each step can be tested alone.

What is an agent, and when is it the right shape?

A system where the model directs its own process and tool use in a loop. It suits open-ended problems where the number of steps cannot be predicted and a fixed path cannot be hardcoded, in an environment where its decisions can be trusted.

Why does the stopping condition matter for architecture?

It sets the worst case. A cap of a given number of iterations bounds the number of model calls a request can use, which bounds latency and cost, and it keeps a wandering loop under control.

What is a compounding error, and how can you see it?

A mistake in an early step carried into later ones. If each of five independent steps is right with probability 0.9, the chain is right with probability about 0.59. Independence is an assumption, but more steps give more chances to go wrong.

How do you decide between a workflow and an agent?

Ask whether the steps can be written in advance. If they can, a workflow gives predictable cost and behaviour; if the path depends on what each step finds, an agent is the shape that fits, with a cap.

Model answer

The question: “We need a system that answers from our documents and takes some actions. What architecture would you propose?” This is the question a design review asks before any code is written. A strong answer hits five beats, in this order:

1. Requirement — state what the system must do and which properties matter: documents, exact identifiers, actions, whether the steps are known.
2. Simplest shape — start with the simplest shape that meets the requirement, from a single call upward.
3. Trade-off — name what the chosen shape adds and locks in: calls, latency, a component to maintain.
4. Failure mode — name how it fails: a retrieval that returns the wrong chunk, or an agent that loops or compounds errors.
5. Flip condition — say what evidence or requirement would move you to a more complex shape.

“I would start from the requirement. If answers must come from documents, I need retrieval with a source to cite. If queries contain exact identifiers such as error codes, I would add lexical search alongside embeddings and consider reranking, accepting two components and an extra call. If actions are needed and the steps are known, I would write a workflow, so the number of calls is fixed and each step is testable. Only if the path depends on what each step finds would I use an agent, and then with a cap on iterations and tools limited to what it needs. I would start with the simplest shape and measure it. The failure to watch for is retrieval returning the wrong chunk, or an agent compounding errors. I would move to the next shape only when a measured failure or a stated requirement called for it.”

Missing a beat is a failure state. Without requirement the shape is chosen by fashion. Without simplest shape the design starts complex. Without trade-off the cost is hidden. Without failure mode the design has no tests. Without flip condition the choice cannot be revisited.

Why this matters

• Design reviews ask for it. A proposal for an agent is judged by whether the path really cannot be written down, and a proposal for a retrieval stack by what it adds.
• Money and latency follow the shape. The number of model calls per request is set by the shape, and that number is multiplied across the traffic.
• Reliability follows the shape. More steps give more chances for an error to carry forward, so the choice affects how the system is tested.
• Talking to customers. When a customer asks why the assistant is not “fully autonomous”, an architect who can say what the extra autonomy would cost in calls, latency and predictability gives a better answer than a refusal.

After this chapter

The project for this chapter is Choose the Shape and Count the Calls (project.md). You will pick a shape for each of eight requests with a printed rule, compute worst-case latency and a chain success probability from stated figures, and finish by writing architecture.tsv: each request with its shape and its maximum number of model calls.

Chapter 3 opens with the architecture.tsv your file holds. The model-call counts in it are the load a system has to be sized and protected for, and Chapter 3 shows how concurrency, quotas, failover and a planned degradation strategy handle that load when capacity runs out.

Hands-on Project — Choose the Shape and Count the Calls

Objective

Pick an architecture for each of eight requests with a printed rule, compute worst-case latency and a chain success probability from stated figures, and write an architecture file, so that the choice of shape rests on a stated rule and stated numbers.

What this project needs: Python 3 and its standard library only. No task needs an account with a cloud or model provider, an API key, an internet connection or any payment. The requests and the figures are printed below, and the figures are invented teaching numbers. All three tasks are required, and Tasks 1 and 2 can be done in either order.

Your evidence file. Keep one plain-text file, chapter2-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print. Where a value is a list, write its items comma-separated with no spaces, in the order given.

The data and the rules for all tasks

Copy this into your Python file. Each request is (id, what is asked, needs_documents, exact_terms, needs_actions, steps_known):

• needs_documents — the answer depends on documents the model was not given in the request;
• exact_terms — the questions contain identifiers (codes, part numbers) where matching the exact word matters;
• needs_actions — the system must call tools or outside systems;
• steps_known — the sequence of steps can be written down in advance.

REQUESTS = [
    ("r1", "Rewrite a paragraph in a friendlier tone",                      False, False, False, True),
    ("r2", "Answer a question from the policy handbook, citing it",         True,  False, False, True),
    ("r3", "Find the fix for an error code in the runbooks",                True,  True,  False, True),
    ("r4", "Look up an order, check the refund rule, draft a reply",        True,  False, True,  True),
    ("r5", "Investigate a failing deployment; checks depend on findings",   False, False, True,  False),
    ("r6", "Label a ticket as billing, technical or other",                 False, False, False, True),
    ("r7", "Answer questions about the catalogue by part number",           True,  True,  False, True),
    ("r8", "Reconcile invoices; which checks run depends on what is found", True,  False, True,  False),
]

The shape rule. For a request, take the first that applies:

1. needs_actions is true and steps_known is false: the shape is agent.
2. needs_actions is true and steps_known is true: the shape is workflow.
3. needs_documents is true and exact_terms is true: the shape is hybrid-rerank.
4. needs_documents is true: the shape is retrieval.
5. Otherwise the shape is single-call.

The figures (invented):

Shape	Model calls per request (worst case)	Lookup steps
single-call	1	none
retrieval	1	one lookup, 0.3 seconds
hybrid-rerank	1	one lookup, 0.3 seconds, and one rerank, 0.5 seconds
workflow	3	none
agent	8, set by its stopping condition	none

Each model call takes 2.0 seconds. A request’s worst-case latency is its model calls times 2.0, plus any lookup steps in the table, all run one after another.

Task 1 — Choose the shape for each request

Write choose_shape(request), which returns one of single-call, retrieval, hybrid-rerank, workflow or agent, following the shape rule. Print the shape for all eight requests. Stop there: eight requests, one rule, and no model is called.

Add to your evidence file:

t1_shapes: <the eight shapes in request order, comma-separated, no spaces>
t1_agent_ids: <ids whose shape is agent, in order, comma-separated, no spaces>
t1_hybrid_ids: <ids whose shape is hybrid-rerank, in order, comma-separated, no spaces>
t1_r4: <shape of r4>

Output: one Python file with a shape function, and four evidence lines.

Task 2 — Latency and compounding error

This task stands on its own: it needs only the figures above.

1. Compute the worst-case latency, in seconds, of one request for each of the five shapes.
2. Suppose each step of a chain is right with probability 0.95 and steps are independent, an assumption made so that the arithmetic is exact. Compute the probability that a chain of 8 steps is right, and that a chain of 3 steps is right.

Add to your evidence file:

t2_latency: <worst-case latency per shape, in the order single-call, retrieval, hybrid-rerank, workflow, agent, comma-separated, no spaces, one decimal place each>
t2_chain_8: <probability for 8 steps, four decimal places>
t2_chain_3: <probability for 3 steps, four decimal places>

Output: the arithmetic, and three evidence lines.

Task 3 — Write the architecture file

This task needs the data, the shape rule and the figures above.

Write architecture.tsv: one line per request, in order, with three fields separated by single tab characters: the request id, its shape, and its worst-case number of model calls from the table.

Add to your evidence file:

t3_rows: <number of lines in architecture.tsv>
t3_total_calls: <the sum of the worst-case model calls over all lines>
t3_agent_calls: <the sum of the worst-case model calls over the lines whose shape is agent>

Output: an architecture.tsv with one line per request, and three evidence lines.

What carries forward

Keep architecture.tsv and chapter2-evidence.txt. Chapter 3 opens with your architecture.tsv. The model-call counts in its third column are the load that a system has to be sized and protected for: how many calls can run at once, what happens when a provider’s limit is reached, and what the system does when capacity runs out.