Chapter 4 — Context Engineering

Course: Getting Started with AI

Prerequisites: Chapters 1 to 3, with the ideas re-defined in the notes before they are used: a context window holds input and output together and is counted in tokens, a prompt is split into a system message, examples and a user message, and rules and examples sent on every request are a recurring token cost. You can read and run a short Python script. No account with a model provider and no API key are needed.

What the reader can do after this chapter:

• Divide a context window into a fixed part, a reserved output allowance and a retrieval budget, and calculate how many retrieved chunks fit in what remains.
• Arrange retrieved chunks so the most relevant ones sit at the start and end of the context, and say what a published study found about position in long contexts and what it does not show.
• Choose between clearing old tool results, compaction, note-taking and a sub-agent for a described long-running task, and name what each choice risks losing.

Core question this chapter answers: When a model’s window is finite and its attention to what is in the window degrades as the window fills, what do you let in, in what order, and what do you do when it starts to fill up?

Connections:

• Builds on: Chapter 3’s system message, examples and user message — the fixed and variable parts of one request, and the token counts you measured for them.
• Leads into: Chapter 5, which takes the reply side of the same request and asks how to make the model’s output machine-usable: a defined shape, validated before anything else uses it.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for an answer in an exact format, and each one is marked by a rule that the format printed with the question lets you check.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines, and Task 3 also from the files it asks you to bring forward. Every accepted value, or the rule that decides it, follows from the data, formulas and rules printed in the project, so you can check your own work before you submit it.

Getting Started with AI — Chapter 4: Context Engineering

Before you start

Nothing to install for the reading. For the project you need Python 3 and the ability to install one package with pip (the same tiktoken package as before). You do not need an account with any model provider, an API key or any payment.

Keep one question in mind while you read: for the request I am about to send, which tokens earn their place, and what happens to the ones that do not? By the end of the chapter you should be able to budget a window, order what goes in it, and say what to do when it starts to fill.

Summary

Context engineering is the work of curating the tokens a model sees at each step: not only the prompt you wrote, but the conversation history, tool results and retrieved documents that land in the window around it. The window is finite, and one provider’s documentation warns that as the token count grows accuracy and recall degrade, so what you let in matters as much as how much room there is. A published study of long contexts also found that models often used information best when it sat at the start or end of the input and worst when it sat in the middle, so order is a design choice too. Budgeting turns this into arithmetic: reserve room for the reply, fix the standing parts, and spend the rest on the highest-value material. When a long task fills the window anyway, four tools are documented: clear old tool results, compact the history into a summary, keep notes outside the window, or hand work to a sub-agent with a clean window.

Key concepts

Term	Definition
Context	Everything the model can reference for one request: the system prompt, every message, tool results, documents, tool definitions and the reply being written.
Context engineering	One provider’s definition: the strategies for curating and maintaining the optimal set of tokens during inference, including information that arrives outside the prompts you wrote.
Context rot	The documented loss of accuracy and recall as the token count in the context grows, even while the context still fits inside the window.
Output reserve	The number of tokens deliberately left free in the window for the model’s reply, because input and output share one limit.
Fixed context	The parts of a request that are the same every time: the system message and the examples.
Retrieval budget	The number of tokens allowed for retrieved material in one request: what remains after the fixed part, the user’s input and the output reserve.
Relevance score	A number that says how well a retrieved chunk matches the request. This chapter treats the scores as given; Build and Evaluate RAG Applications covers how they are produced.
Position effect	A difference in how well a model uses information depending on where in the context it sits, for example at the start, in the middle or at the end.
Tool-result clearing	Replacing older tool results in the history with a short placeholder once the model has used them, so they take only the placeholder’s few tokens.
Compaction	Taking a conversation that is nearing the window limit, summarizing it, and starting a new window with the summary.
Structured note-taking	The agent writes notes to storage outside the window and reads them back in later.
Sub-agent	A separate model run, with its own clean window, that does a focused piece of work and returns only a short summary of it to the main run.

Explanations

The failure, before the explanation

A support assistant answers questions about a product’s documentation. For each question the team retrieves the 20 best-matching chunks of documentation and pastes all of them into the request, then adds the whole conversation so far. In testing, with short conversations, it was accurate. In production a customer asks about a refund rule in the sixth message of a long chat, and the answer is wrong, even though the chunk with the correct rule was among the 20. The team’s first thought is that retrieval failed. It did not: the right chunk was in the window. The failure is in what surrounded it and where it sat.

This need not be a fault in the model’s ability to read. A finite window, a signal that can degrade as the window fills, and no curation can add up to exactly this. The rest of the chapter takes the pieces apart: what counts as context, why more is not better, why order matters, how to budget, and what to do when the window fills.

What counts as context, and what context engineering adds to prompt engineering

Open one of the support assistant’s requests and tally its window. Instructions and examples, 1,200 tokens. The customer’s latest message, 80. Twenty retrieved chunks, 9,000. Earlier turns of the chat, 6,000. Tool definitions, 700. The reply being written, 400 so far. These figures are invented for illustration, but the list of what is there is not. One provider’s documentation says everything in the request counts toward the window: “the system prompt, every message in messages (including tool results, images, and documents), and your tool definitions”, and “the output Claude generates for the turn, including its extended thinking, counts too” (Anthropic, Context windows). Chapter 3’s prompt parts are only the first 1,200 tokens of that tally; the history, every tool result and every retrieved document are tokens in the same window.

The same provider’s engineering team draws the line between the two crafts. Prompt engineering “refers to methods for writing and organizing LLM instructions”; context engineering “refers to the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference, including all the other information that may land there outside of the prompts”. They add that for an agent running in a loop, “the curation phase happens each time we decide what to pass to the model” (Anthropic, Effective context engineering for AI agents). A prompt is written once; context is decided on every call.

Picture the window as a desk. The instructions and the question are two sheets laid out first. Then the history, the search results and the tool outputs arrive as files, one after another. The desk is a fixed size, and so is your attention: a desk with ten relevant files is one you can work from; a desk with three hundred files is one where the file you need exists somewhere, and finding it is the job.

A second way to see it: the window is the working memory of a program that cannot page anything in or out by itself. Anything you want the model to use has to be loaded into it by your code, and anything you do not unload stays there, taking up room, until you remove it. Context engineering is the memory management.

Why more is not better

Take the support assistant and ask it the same refund question twice: once with only the three chunks that bear on the question, and once with all twenty chunks plus the whole chat. The model is the same and the right chunk is present both times; the second window is larger and noisier. Whether the second answer is worse is something you would measure, but the documentation says to expect that it can be. Chapter 1 introduced the name for this, context rot. Here is the statement in full from the context-window documentation: “A larger context window allows the model to handle more complex and lengthy prompts, but more context isn’t automatically better. As token count grows, accuracy and recall degrade, a phenomenon known as context rot. This makes curating what’s in context just as important as how much space is available” (Anthropic, Context windows).

The engineering post explains why and says how far the effect goes. It describes an “attention budget”: the model’s ability to relate each token to every other token is “stretched thin” as the context grows. It adds that these factors “create a performance gradient rather than a hard cliff: models remain highly capable at longer contexts but may show reduced precision for information retrieval and long-range reasoning compared to their performance on shorter contexts”. Its conclusion for design: “good context engineering means finding the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome” (Anthropic, Effective context engineering for AI agents).

Read the claim at its real size. It does not say a long context always fails, and it does not give a token count where quality falls off. It says quality can fall gradually as the context grows, that different models degrade at different rates, and that the safe design is to put in only what the task needs. “Minimal” is not the same as short: the post itself says you still need to give the agent enough information up front to follow the behaviour you want.

Applied to the support assistant: twenty chunks for every question is not a design, it is a default. Whether the sixth-message refund question needs twenty chunks, or four, is something you measure; but the starting position should be the smallest set that could answer, not the largest that fits.

What you see	Likely cause	What to do instead
Answers get worse as a chat grows, and the right chunk is in the window	The window is crowded with material that is not needed for this question	Include only what this question needs; measure with and without the extra material
A request is accepted but the answer is vague on a detail buried in a long input	Fitting in the window is not the same as being used well	Extract the relevant part before sending it
The team plans to “just wait for a bigger window”	The engineering post expects context windows of all sizes to face context pollution and relevance concerns for the foreseeable future, where the strongest performance is wanted	Treat context as a budget at any window size

Order: where in the window something sits

Next, position. A study first posted as a preprint in July 2023 tested language models on tasks that need a model to find relevant information in its input: multi-document question answering and key-value retrieval. Its finding: “performance can degrade significantly when changing the position of relevant information”, and “performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models” (Liu et al., Lost in the Middle: How Language Models Use Long Contexts).

Keep the finding at its real size too. It was first posted in 2023, so it tested the models of that time, on two kinds of task. It does not show that every current model has a “middle” blind spot, and the engineering post describes the broader effect as a gradient rather than a cliff. What it does give you is a reasonable default and a thing to test: if your task depends on one or two key passages, where they sit may matter, so try them at the start and end and compare.

One provider gives practical advice that fits. For long inputs of 20,000 tokens or more, its guide says to “place your long documents and inputs near the top of your prompt, above your query, instructions, and examples”, and reports that “queries at the end can improve response quality by up to 30 percent in tests, especially with complex, multidocument inputs” (Anthropic, Prompting best practices). The guide’s “in tests” and “can” are doing work: it is a result on that provider’s tests, not a promise for your task.

Here is a simple arrangement rule that applies the study’s finding, which this chapter’s project uses. It is an exercise rule, not something either source prescribes. Rank the chunks by relevance, 1 being the best. Put rank 1 first and rank 2 last, rank 3 second and rank 4 second-to-last, and so on, so that the highest-ranked chunks are at the two ends and the lowest-ranked are in the middle. Five chunks ranked 1 to 5 come out in the order 1, 3, 5, 4, 2.

A second way to picture the arrangement rule: a stack of sheets handed to a reviewer, with the two sheets you most want read placed on top and at the bottom, and the least important ones in the middle. That is a picture of the rule, not a claim about how people or models read.

Ordering does not stand alone. The fixed part (system message and examples) still goes first and stays identical on every request so that Chapter 2’s cache can reuse it. The ordering rule applies to the variable material inside the window: the chunks.

Budgeting: arithmetic for a finite window

Start with an example. A window holds 8,000 tokens (a round teaching figure, not any model’s). The fixed part is 1,200 tokens, the user’s input is 300 and the output reserve is 1,000. The retrieval budget is what is left: 8,000 − 1,200 − 300 − 1,000 = 5,500 tokens. With chunks of 400 tokens each, 13 chunks fit (13 × 400 = 5,200; a fourteenth would need 5,600).

The general form is four numbers that must add up to no more than the window:

window ≥ fixed part + user input + retrieval budget + output reserve

Why a reserve? Input and output share one limit. One provider’s API rejects an input that already exceeds the window with an error, and on its newer models a response that reaches the window limit stops with the stop reason model_context_window_exceeded (Anthropic, Context windows). The same page notes that cached prompt prefixes still occupy the window: caching changes what you pay for those tokens, not whether they count. A request that spends the whole window on input leaves nothing for the answer.

Why a budget rather than “as many as fit”? Because of the previous section: the extra chunks are not free. Fitting is a limit; a budget is a decision below the limit. The numbers above say 13 chunks can fit; whether 13 should is the question context rot raises, and the answer is something to test.

When chunks are not all the same size, the same arithmetic needs a rule. This chapter’s project uses a simple one: go through the chunks from the most relevant down, add each one that still fits, skip any that does not, and carry on. A chunk that does not fit is skipped rather than ending the search, because a smaller, lower-ranked chunk may still fit in the space left.

What you see	Likely cause	What to do instead
An API error that the input exceeds the window	No output reserve and no cap on retrieved text	Compute the budget before the call; trim retrieval to fit
Replies cut off at the output ceiling	The window was nearly full, so the reply had little room	Reserve output tokens before adding retrieved material
Lower-ranked chunks crowd out nothing but add noise	The budget is “everything that fits”	Set the retrieval budget below the limit and test the answer quality

When the window fills anyway: four documented tools

Long tasks fill windows whatever the budget at the start. An agent that calls tools in a loop adds a tool result to the history on every step. Four tools are documented: the engineering post names compaction (with tool-result clearing as its lightest form), structured note-taking and sub-agent architectures, and the provider’s documentation covers clearing and compaction as features. Each is a different trade.

Clear old tool results. Once a tool’s raw result has been used, the model rarely needs to see it again. The post calls clearing “one of the safest lightest touch forms of compaction”. The provider’s context-editing feature does this by rule: when the context passes a threshold, it clears the oldest tool results “in chronological order” and replaces each with placeholder text that tells the model it was removed (Anthropic, Context editing). It loses the raw detail of old results; it keeps the conversation intact.

Compact the history. Compaction “is the practice of taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary” (Anthropic, Effective context engineering for AI agents). The provider documents it as a feature that replaces the older turns with a summary written by the model, and notes that recent turns can be kept word for word (Anthropic, Compaction overview); another provider documents a threshold-based version that emits an opaque “compaction item” carrying the state forward (OpenAI, Compaction). The risk is stated plainly: “overly aggressive compaction can result in the loss of subtle but critical context whose importance only becomes apparent later”. The post’s advice is to tune the compaction prompt on real agent traces, first maximizing recall and then improving precision.

Keep notes outside the window. In structured note-taking, “the agent regularly writes notes persisted to memory outside of the context window”, which “get pulled back into the context window at later times”; the post’s examples are a to-do list or a NOTES.md file that tracks progress across a long task (Anthropic, Effective context engineering for AI agents).

Hand work to a sub-agent. A sub-agent works in a clean window and “returns only a condensed, distilled summary of its work (often 1,000-2,000 tokens)”, so the detailed search stays out of the main window (same source).

The post gives a rule of thumb for choosing, and gives it as examples, not as a law: “Compaction maintains conversational flow for tasks requiring extensive back-and-forth; Note-taking excels for iterative development with clear milestones; Multi-agent architectures handle complex research and analysis where parallel exploration pays dividends.”

Tool	Best when	It is the distractor when	What it risks
Clear old tool results	Many large tool outputs that have already been used	The old results are still needed verbatim	Losing raw detail the model might need again
Compaction	Long back-and-forth where continuity matters	Early details are precise and critical	A summary that drops a detail whose importance shows up later
Notes outside the window	Iterative work with clear milestones	The state cannot be written as short notes	Notes that are incomplete or never read back
Sub-agent	A side task that would dump much material into the window	The side task needs the whole conversation	The summary it returns leaves something out

A second way to see these: they are the four ways to clear a desk. Throw away the files you have finished with (clear results), replace the pile with a one-page summary (compact), write the state of play on a sticky note kept elsewhere (notes), or send the file-hunting to a colleague who comes back with a page (sub-agent).

One more option is mentioned by the engineering post: not loading everything up front. Rather than pre-loading all relevant data, an agent can keep “lightweight identifiers (file paths, stored queries, web links, etc.)” and load data into context at runtime with tools, which the post calls a “just in time” approach. That trades a smaller window for more tool calls, and Build and Evaluate RAG Applications picks it up.

Three situations, and what they mean for your design

Before the table, three situations:

1. A document-question service pastes the 25 best chunks into every request. Answers to questions about a specific clause are sometimes wrong when the clause is a middle chunk.
2. An agent working through a large codebase migration reads many files, and its history is mostly raw file contents it has already used. The window is 85% full after many tool calls.
3. An agent running a multi-stage task with clear milestones loses track of which stage it is on after its history is compacted.

What you see	Which lever	What to do
A buried clause is missed (situation 1)	Budget and order	Cut to the chunks the question needs, arrange so the best sit at the start and end, and test with the key chunk in different positions
History mostly raw, already-used tool output (situation 2)	Tool-result clearing	Clear the oldest tool results by rule; keep the recent ones
State is lost across compaction (situation 3)	Notes outside the window	Write stage and decisions to a notes file the agent reads back after compaction; tune the compaction prompt on real traces

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

What does context include, beyond the prompt you wrote?

Everything in the request: the system prompt, every message (including tool results, images and documents), tool definitions, and the output being generated. History and retrieved material take up the same window as your instructions.

How does one provider distinguish context engineering from prompt engineering?

Prompt engineering is writing and organizing instructions. Context engineering is curating and maintaining the optimal set of tokens during inference, including information that arrives outside the prompts, and it is decided again on every call.

What is context rot, and does it mean long contexts always fail?

It is the documented loss of accuracy and recall as the token count grows. It does not mean failure at a fixed size: the same source calls it a performance gradient rather than a hard cliff, and says models degrade at different rates.

What does "smallest possible set of high-signal tokens" mean in practice?

Include what the task needs and nothing that merely fits. “Minimal” is not “short”: you still give the model enough information to follow the behaviour you want.

What did the "Lost in the Middle" study find?

On multi-document question answering and key-value retrieval, performance was often highest when the relevant information was at the beginning or end of the input and degraded when it was in the middle, including for models built for long contexts. It tested models of its time (the preprint dates from 2023) on two task types.

What four numbers make up a context budget?

The fixed part (system message and examples), the user’s input, the retrieval budget and the output reserve. Together they must not exceed the window.

Why reserve tokens for the output?

Input and output share one limit, so a window filled with input leaves no room for the reply and the response can be cut off.

A retrieved chunk does not fit in the space left, but a smaller lower-ranked one would. What does the chapter's packing rule do?

It skips the chunk that does not fit and carries on down the ranking, so the smaller chunk can still be added.

What does tool-result clearing do, and what does it lose?

It replaces older tool results with placeholder text once a threshold is passed, so each takes only the placeholder’s few tokens instead of its full length. It loses the raw detail of those results, which matters only if the model needs them again.

What is the main risk of compaction?

A summary that is too aggressive can drop subtle but critical context whose importance only shows up later. The documented advice is to tune the compaction prompt on real traces, maximizing recall first.

When does the engineering post suggest each of compaction, notes and sub-agents?

Compaction for tasks with extensive back-and-forth, note-taking for iterative development with clear milestones, and multi-agent designs for complex research where parallel exploration pays off. It gives these as examples of the choice, not as fixed rules.

Model answer

The question: “Our assistant’s answers get worse as a conversation gets longer, and the right document is in the context when it goes wrong. What is happening, and what would you change?” This is the question a lead engineer asks after the failure above. A strong answer hits five beats, in this order:

1. Budget — the window is finite, input and output share it, and a budget is set below the limit.
2. Selection — more context is not better: context rot means include the smallest set that could answer.
3. Order — position can matter, so the key material goes at the start and end, and that is tested.
4. Reduction — when the window fills: clear old tool results, compact, keep notes, or use a sub-agent.
5. Verification — measure the effect with and without the change, and check what compaction dropped.

“The right chunk being in the window doesn’t mean it was used well. The window is finite and shared with the reply, so I’d put a budget on it: fixed part, the user’s input, an output reserve, and whatever is left for retrieval, and I’d set that retrieval budget below the limit, because the documentation says accuracy and recall degrade as the token count grows. So I’d include the smallest set of chunks that could answer rather than the twenty that fit. Order also matters: a study found that models often did best with relevant information at the start or end of the input and worse in the middle, so I’d put the highest-ranked chunks at the two ends, while noting that it tested older models. As the chat grows I’d reduce the history: clear old tool results first, because that is the lightest touch, compact the conversation if it still fills, keep milestone state in notes outside the window, and push side searches to a sub-agent. Each loses something, so I’d check by running the same questions with and without the change, and by testing what a compacted history no longer contains.”

Missing a beat is a failure state. Without budget the window has no plan and the reply has no room. Without selection the answer treats fit as sufficient. Without order it ignores a documented position effect. Without reduction it has nothing to say when the window fills. Without verification every claim above is a theory — and that is the beat reviewers most often find missing.

Why this matters

• Quality in production. A system that is accurate in short tests and drifts in long sessions may have a context problem; other causes are possible, so compare the same questions with a short and a long history before deciding. Budgeting and curating are the levers this chapter gives you for it.
• Cost and latency. Each turn’s input “contains all previous conversation history plus the current user message”, and that history is “billed as input tokens” (Anthropic, Context windows), so context you keep is paid for again on every later call. A tight context is cheaper; on latency, the guide cited in Chapter 2 puts the gain from halving a prompt at only 1–5%, so expect the saving mostly in cost (OpenAI, Latency optimization).
• Agent reliability. Agents add to their context on every step. How a system clears, compacts or offloads that history is one of the things that affects whether a long task stays coherent.
• Design reviews. “We paste in the top 20” is a default. A stated budget with a reason for each number is a design a reviewer can challenge.

After this chapter

The project for this chapter is Budget, Order and Shrink a Context (project.md). You will pack retrieved chunks into a budget and arrange them, reduce a long tool-heavy history by clearing and compaction, and finish by budgeting the request you built in Chapter 3.

Chapter 5 starts from the same request and turns to its reply. It asks how to get output a program can use: a defined shape, validated at the boundary, and what to do when the model returns something malformed.

Hands-on Project — Budget, Order and Shrink a Context

Objective

Pack retrieved chunks into a token budget and arrange them, reduce a long tool-heavy history by clearing and compaction, and finish by budgeting the request you built in Chapter 3, so that you can say exactly what goes into a window, in what order, and what is done when it fills.

What this project needs: Python 3 with the standard library only for Tasks 1 and 2. Task 3 counts tokens and so also needs the tiktoken package (pip install tiktoken, which needs an internet connection). The package, its cl100k_base encoding and the enc.encode call are the ones in the OpenAI Cookbook, How to count tokens with tiktoken. No task needs an account with a model provider, an API key or any payment, and no task calls a model. All three tasks are required.

Your evidence file. Keep one plain-text file, chapter4-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print. A list value is written as items separated by commas with no spaces.

Task 1 — Pack and arrange retrieved chunks

A request has a window of 2,000 tokens. Reserve 300 tokens for the output. The fixed part is the system message (220 tokens), the examples (180 tokens) and the user’s input (60 tokens), which is 460 tokens in all. Six retrieved chunks are available, each with a token count and a relevance score. Copy this data exactly:

CHUNKS = [
    ("A", 400, 0.91),
    ("B", 350, 0.88),
    ("C", 500, 0.84),
    ("D", 300, 0.80),
    ("E", 250, 0.77),
    ("F", 200, 0.70),
]

Write two functions using only the standard library.

1. pack(chunks, window, reserve, fixed) — the retrieval budget is window - reserve - fixed. Go through the chunks from the highest relevance score to the lowest. Add a chunk if its token count is at most the budget that is still left; otherwise skip it and carry on with the next chunk. Return the budget, the included chunk ids, the skipped chunk ids (both in relevance order), the tokens used by included chunks and the budget left unused.
2. arrange(ids) — takes chunk ids already in relevance order, best first, and returns them in this order: the 1st, 3rd, 5th, … go from the front in that order; the 2nd, 4th, 6th, … go in from the back, so the 2nd-ranked id is last. For five ids ranked 1 to 5 the result is 1, 3, 5, 4, 2. This is an exercise rule that puts the highest-ranked items at the two ends; it is not a rule that any source prescribes.

Then print, for CHUNKS: the budget, the included ids, the skipped ids, the tokens used and the budget left; the result of arrange on the included ids; and the result of arrange on all six ids in relevance order. Finally, consider a different rule that you do not keep in your file: if packing stopped at the first chunk that did not fit instead of skipping it, which ids would be included? Work it out from the same data and record it too.

Add to your evidence file:

t1_available: <the retrieval budget>
t1_included: <included ids, relevance order>
t1_skipped: <skipped ids, relevance order>
t1_chunk_tokens: <tokens used by included chunks>
t1_unused: <budget left unused>
t1_order_included: <arrange() of the included ids>
t1_order_all: <arrange() of all six ids>
t1_stop_at_first_miss_included: <ids included if packing stopped at the first miss>

Output: one Python file with two functions, and eight evidence lines. Stop there: two functions, not a retrieval system.

Task 2 — Shrink a long history

An agent’s history has ten turns. Each has a role, a token count and a kind: m for an ordinary message, t for a tool result. Copy this data exactly:

HISTORY = [
    ("user", 120, "m"),
    ("assistant", 200, "m"),
    ("tool", 900, "t"),
    ("assistant", 150, "m"),
    ("user", 80, "m"),
    ("tool", 700, "t"),
    ("assistant", 180, "m"),
    ("user", 90, "m"),
    ("tool", 800, "t"),
    ("assistant", 160, "m"),
]

Write two functions using only the standard library.

1. clear_tool_results(history) — keep the most recent tool result at its full size. Every older tool result is replaced by a placeholder that counts as 12 tokens. Messages are unchanged. Return the new history.
2. compact(history, threshold, summary_tokens, keep_last) — if the history’s total is more than threshold, replace every turn except the last keep_last turns with one summary turn of summary_tokens tokens, placed first; otherwise return the history unchanged.

Run them in this order, with a threshold of 1,500, a summary of 250 tokens and the last 3 turns kept: clear tool results from HISTORY, then compact the result. Print the total of HISTORY, the total after clearing, the tokens saved by clearing, whether the cleared total is over the threshold, and the total after compaction.

Add to your evidence file:

t2_total_before: <number>
t2_after_clearing: <number>
t2_saved_by_clearing: <number>
t2_over_threshold: <yes or no>
t2_after_compaction: <number>

Output: one Python file with two functions, and five evidence lines.

Task 3 — Budget your own request

Use the three files you wrote in Chapter 3: system.txt, examples.json and user.txt. If you do not have them, make stand-ins now: a system.txt of three to six lines of rules, an examples.json that is a JSON list of three objects each with input and output strings, and a user.txt holding one input wrapped in <input> and </input>. Only the token counts are marked, so any content will do.

1. Count tokens with tiktoken.get_encoding("cl100k_base") and len(enc.encode(text)), on the contents exactly as open(name, encoding="utf-8").read() returns them: system.txt; user.txt; and the examples, counting the input string and the output string of every example separately and adding the counts.
2. Assume a window of 8,192 tokens (a round teaching figure, not any model’s window) and an output reserve of 1,000 tokens.
3. The fixed tokens are system + examples + user. The retrieval budget is the window minus the reserve minus the fixed tokens. If it comes out below zero, write that negative number as the budget and use 0 below.
4. Retrieved chunks are 400 tokens each. Work out how many whole chunks fit in the budget, the tokens those chunks use, and the budget left unused after them.
5. Work out the fixed tokens as a share of the window, to 3 decimal places.

Add to your evidence file:

t3_system_tokens: <number>
t3_examples_tokens: <number>
t3_user_tokens: <number>
t3_fixed_tokens: <system + examples + user>
t3_available: <8192 − 1000 − fixed>
t3_chunks_that_fit: <whole 400-token chunks that fit, 0 if the budget is negative>
t3_chunk_tokens_used: <chunks that fit × 400>
t3_unused_tokens: <budget − chunk tokens used, or 0 if the budget is negative>
t3_fixed_share: <fixed ÷ 8192, 3 decimal places>

Output: a worked budget for your own request, and nine evidence lines. Submit the three files with the evidence file so the counts can be re-checked.

What carries forward

Keep system.txt, examples.json, user.txt and chapter4-evidence.txt. The next chapter opens with the same request and asks what shape its reply should have: you will define that shape and write the check that decides whether a reply meets it.