Courses / Agentic AI from Scratch (Python)

Session 2 of 40

How an LLM works: from rules to transformers, then tokens, the context window and cost

Notes

How Does an LLM Work? Foundation Models, Transformers, Tokens, the Context Window and Cost Explained

Agentic AI Foundations · How an LLM works: from rules to transformers, then tokens, the context window and cost

Before you start: what do you need to learn how a large language model works?

Nothing to install to read the explanations. To do the small tasks you need what you set up in the first session (Python, the course code folder and a working model endpoint) and, for the token counter, one extra package installed with pip install tiktoken. Two free websites help too: a tokenizer page and a price calculator, named in the cost section. Keep the first-call reply you saved in notes/first-call.md. Keep paper open for the test set and for the flashcards, which ask you to say answers aloud. You need no maths: the transformer is explained as a picture.

Summary: how does an LLM read text, and what does a call cost?

This session opens up the first call. It starts with a short history of how software came to write text: hand-written rules, machine learning, deep learning, and today’s foundation models that one prompt steers to many tasks. Next come tokens, the small pieces a model reads and writes one at a time, and a five-step picture of how a chat reply is built from a question. Attention, the idea inside that picture, shows why a model can use a word anywhere in a text, and the transformer is the machine built by stacking many attention steps. The context window limits how many tokens fit in one request. The session ends with the cost of a call, an estimate you can make before sending, four small programs that show real usage and how it grows as a conversation grows, and the test set of ten questions that will judge later changes to the Copilot.

Key concepts: foundation model, attention, token, context window and cost defined

Term Definition
Foundation model One model trained on broad data at scale that can be adapted to a wide range of tasks; a large language model is one.
Machine learning Building a model that finds patterns in data and does a task without explicit instructions, instead of following rules a person wrote out.
Deep learning A family of machine learning algorithms made of several layers of artificial neural networks.
Token The small piece of text a model reads and writes: a whole word, part of a word or a punctuation mark.
Next-token prediction Picking a likely next token given the tokens so far, adding it, and repeating until the reply is complete.
Attention For the word being worked on, scoring the other words for how much they matter and mixing in more of the highly scored ones.
Transformer A model built from many attention layers stacked one on another; in each layer the tokens look at the other tokens and update their meaning.
Context window The most tokens one request can use, counting what you send, what the model writes and its reasoning tokens.
Cost of a call The input tokens and the output tokens, each multiplied by the provider’s price per million tokens.
Test set A fixed list of questions, each with the answer you expect, run again whenever a prompt or model changes.

Explanations: how do foundation models, attention, tokens, the context window and cost fit together?

Where are we? From a first call to knowing what a call costs

In the first session you sent a message to a model and read the reply. It may have felt like plain text in, plain text out. It is more than that: a call has a size and, on a hosted service, a price, and both are measured in the same unit. The new parts of the Copilot this session are a token counter with a cost estimate, and a test set of ten questions. Already built: the first call from the first session.

How did AI get from rules to foundation models and large language models?

Take one question: “How many leave days can I carry over?” Here is how software has handled it over time.

Era How the question is handled The limit
Rules If the message contains “leave”, send paragraph 4 Breaks on any phrasing nobody listed
Machine learning A model learns from examples to label messages as leave, laptop or expenses Usually retrained with new labelled examples for each new task
Deep learning A neural network with several layers learns from text and images Models of this era were usually built and trained for one job each
Foundation model One model trained on broad data, steered by a prompt to label, summarise, draft or answer Can be wrong in many tasks at once

The first row is a receptionist with a card index: fine until someone phrases the question a way the card does not list. A cloud provider’s guide describes machine learning as performing data analysis tasks without explicit instructions: the model finds patterns in data instead of being told the rules. Deep learning, in another of its guides, is a family of machine learning algorithms made of several layers of artificial neural networks.

The step after that gave the field a new name. The research report that coined the term described foundation models as trained on broad data at scale and adaptable to a wide range of downstream tasks. It also warned that the defects of a foundation model are inherited by the models adapted from it. Picture a clerk who learned one form letter, replaced by a clerk who can draft any letter: the range is the gain, and one bad habit now shows up in the letters it drafts. A second picture is a Swiss army knife next to a drawer of single-purpose tools: with a foundation model you choose the blade with the prompt. A large language model is a foundation model for text. This is why the same model in the Copilot can answer a question now and, in a later session, sort a question into a category, with just the instructions changed.

What is a token and how does next-token prediction write a reply?

Imagine a Lego model. Short, common words are one brick; a rare word is built from several small bricks. A model reads and writes text in bricks like these, called tokens: a whole word, part of a word or a punctuation mark. OpenAI’s documentation describes tokens as commonly occurring sequences of characters. A second picture is postage by weight: a parcel is charged by what is on the scale, not by how many letters are on the label, and a model call is charged by tokens, not words.

A word is not a token. Here is the same sentence as words and as pieces. The split is illustrative, written for these notes; a real tokeniser makes its own splits.

 Sentence:  Up to 5 unused days carry over.

 Words:     Up | to | 5 | unused | days | carry | over.
 Pieces:    Up | to | 5 | unused | days | carry | over | .
 A rare word:   reimbursement  ->  reim | burse | ment

For English text, OpenAI gives a rough rule of thumb: one token is about 4 characters, or about 0.75 words. It is a rough rule; for a real figure, count the text itself.

Now how a reply is written. On your phone, autocomplete suggests the next word, and you can keep tapping the suggestion. A language model does this one token at a time. This is next-token prediction: given the tokens so far, it picks a likely next token, adds it to the text, and picks again until the reply is complete. For “Up to 5 unused …”, “days” is a likely next token, “leave” is a less likely one, and “bananas” is very unlikely.

 Question tokens in  ->  [model]  ->  "Up"
 Question + "Up"     ->  [model]  ->  "to"
 Question + "Up to"  ->  [model]  ->  "5"
 ...  repeats until the reply is complete

Two things follow. Tokens you send in are counted, and tokens the model writes out are counted too. A longer reply is more tokens written.

How is a chat reply built? From a question to an answer, step by step

Think of a clerk answering a staff question at a front desk. The clerk hears the question, pulls the right page from the policy folder, reads the page against the question, and writes back. A model call has the same five steps, and this picture is the whole thing the Copilot will do. It builds on the tokens from the section above.

Flow of a chat reply: an employee asks how many leave days can be carried over; in your program, before the model, step 1 picks the policy section and step 2 builds the request; inside the model, step 3 reads it all as tokens with attention, and step 4 writes the reply token by token, each pick a draw set by temperature; step 5 is the answer, either up to 5 days used by 31 March, or not in the policy, raise a ticket.

The same five steps as text:

 Employee: "How many leave days can I carry over?"
        |
        v
 1. Pick the policy section      leave policy, 4.2 "Carry-over"
        |
        v
 2. Build the request            role and tone + section + question
                                  settings: model, temperature
        |
        v
 3. Read it all as tokens        attention links the question to the section
        |
        v
 4. Write the reply, token by token   each pick is a draw (temperature)
        |
        v
 5. Answer "Up to 5 days, used by 31 March"   or   "Not in the policy: raise a ticket"

In the picture, steps 1 and 2 are your program, before the model; steps 3 and 4 happen inside the model; step 5 is what the employee reads. Read it from the top. Step 1 picks the policy section that holds the answer, here the carry-over paragraph of the leave policy. In this session you do step 1 by hand: you choose the section yourself. A later module, retrieval, does it automatically. Step 2 builds the request: the role and tone you want, the section and the question, plus the settings, which are the model and the temperature. Step 3 reads the whole request as tokens; attention, explained in the next two sections, is how the model links the question to the section. Step 4 writes the reply one token at a time, and each pick is a draw, which is where temperature comes in. Step 5 is the answer: either the figure from the section, or a note that the policy does not say and a ticket should be raised.

A second way to see it is a recipe card: the section and the question are the ingredients, the settings are the oven dial, and the reply is the dish. Because step 4 is a draw, the same request can come out slightly differently, so the answer can change with the settings. A model works from what is in the request: if step 1 picks the wrong section, even a good reply is built on the wrong page.

Why does a word need the words around it?

A word on its own is like hearing one sentence of a phone call from the next room: you catch the words but not what they refer to. Take “bank”. In “I sat on the river bank” it is a riverside; in “I paid the cash in at the bank” it is a place that holds money. The other words decide.

A sharper pair is a classic example. Read both:

 The laptop did not fit in the bag because it was too big.
                                              it = the laptop
 The laptop did not fit in the bag because it was too small.
                                              it = the bag

Just the last word changed, and yet “it” now points to a different noun. A model that read left to right, one word at a time, without looking around, could not tell which noun “it” points to. So each word needs a way to look at the rest of the text. That way is attention.

What is attention?

Attention is how a model, for the word it is working on, scores the other words for how much they matter and mixes in more of the highly scored ones. Picture a reader with a highlighter who marks the words that matter to the one being read. Or a meeting where each person listens hardest to the colleague who knows about their topic. Here is the laptop pair as weights, written for these notes; they are illustrative, not values from a real model:

 "...because it was too big"   "...because it was too small"
   "it" looks at:                "it" looks at:
     the laptop ======  strong     the laptop - - -   weak
     the bag    - - -   weak       the bag    ======  strong
 (illustrative weights, not from a real model)

The same idea works in the Copilot. Take “How many leave days can I carry over?” sent together with policy section 4.2. The words “carry over” attend strongly to “Carry-over” in the section, so the model finds the figure. That is step 3 of the reply picture: the model reads the request as tokens and works out which parts relate to which. Attention is small and concrete: score the other words, then blend in the ones that matter.

What does attention mean in an LLM? A city example

Think of a traffic control room in a big city. One operator watches dozens of camera feeds. The operator does not stare at every feed equally. When an ambulance is on its way, the operator looks hardest at the junctions on its route and glances at the rest. Attention in a model works like that: for the word it is working on, it looks hardest at the few words that decide what this word means.

Here are two everyday sentences where a word becomes clear once you look at the others. The weights are illustrative, written for these notes; they are not values from a real model:

 "The bus to the airport was late, so IT left stop 6."
   it -> the bus      =========  strong
   it -> the airport  - - -      weak

 "Turn left at the hospital, then park behind IT."
   it -> the hospital =========  strong
   it -> the left turn  - - -    weak
 (illustrative weights, not from a real model)

In the first sentence, “it” is the bus, not the airport: the bus is what was late and what left. In the second, “it” is the hospital, because that is the building you park behind. A person does this without noticing. The model does the same thing with scores: it gives each other word a weight, then blends in more of the words with high weights. A second way to see it is a reader with a highlighter, marking the words that matter to the one being read.

In the Copilot, the same step runs on “How many leave days can I carry over?” together with section 4.2. The words “carry over” look hardest at “Carry-over” in the section, which is how the model finds the figure.

What is a transformer?

A transformer is a model built from many attention layers stacked one on another. In each layer the tokens look at the other tokens at the same time and update their meaning. After many layers, the last position predicts the next token, which is the next-token prediction from the token section.

 tokens -> [attention layer] -> [attention layer] -> ... -> next token
            each token looks      each token looks
            at the others         at the others

Think of an editing team reading a draft in rounds. In each round the editors read the whole page and refine their notes; after many rounds they agree what comes next. An older kind of model was more like a person reading through a keyhole, one word at a time. The paper that introduced this design proposed a network architecture, the Transformer, built solely on attention and dropping the older recurrence and convolution steps. You do not need the maths to build with models. You need the order of ideas: attention is the step; the transformer is the machine made of many such steps, which is why this page teaches attention first.

Where are transformers used in real life?

You have met the machine already, even if you did not call it by that name. When you translate a menu, each word looks at the others, so a dish name stays together and comes out as one dish. When you summarise a long email thread, attention can link “the deadline” in message 40 back to the project named in message 1. When your phone suggests the next words in a message, the layers have read what you typed so far, and the last position offers likely next words. The chat assistant from session 1 is the same machine: your question goes in, and the reply comes out one token at a time.

The jobs differ, but the shape is the same: tokens go in, many attention layers read them, and the next token comes out. A kitchen is a fair picture: very different dishes, one set of ovens and benches.

How does a transformer work on one sentence? A walk through the layers

Take a sentence that stops half way: “The bus to the airport leaves from the stop near the station, so catch it at …”. Here is how a stack of layers might get from this sentence to a guess for the next word. This is a teaching picture, written for these notes and not taken from a real model:

 "The bus to the airport leaves from the stop near the
  station, so catch it at ..."        (illustrative)

 Early layers    link nearby words: "the airport"+"bus"
 Middle layers   settle "it": it = the bus
 Later layers    see that a clock time comes next
 Last position   holds all of it -> odds: 7 (60%) ...

Read it as a pipeline. Each layer reads what the layer before it produced, so many layers in a row can turn a bare sentence into a prediction. Think of a relay of editors again: the first fixes the neighbouring words, the next works out what “it” means, and the last sees that “at” wants a clock time. The odds at the end are invented for the example. Real layers do not split the work this neatly, so use this picture to understand the order of ideas, not to say what any one layer learns.

How do attention and the layers pick the next word? The loop in one picture

Now put attention and the layers together. This picture follows the same bus sentence through one full pick of the next word, and then round again:

A loop: the sentence so far goes into layer 1, where the words look at the other words, then through more layers until the last position holds the meaning; the model gives illustrative odds for the next word, 7 at 60 percent, 8 at 25 percent and noon at 10 percent; one word is picked and added to the sentence, and the loop repeats for the next word.

Read it from the top. The sentence so far goes in. In layer 1, attention lets each word look at the others. More layers follow, and the meaning builds up until the last position holds it. From there the model gives odds for the next word: 7 at 60 percent, 8 at 25 percent and noon at 10 percent. These figures are illustrative. One word is picked, here 7, and added to the sentence. Then the whole thing runs again for the next word, which is what the dotted line shows.

This loop is the whole of steps 3 and 4 of the reply picture earlier on this page: step 3 reads the request with attention and layers, step 4 picks one token, and the reply is built by going round again and again until it is complete. What to remember: attention is the step, the layers repeat it, and one pick at the end is added and fed back in. The next session explains how that pick is made.

What is a context window, and how much text fits in one request?

Picture a desk of fixed size. A limited number of pages fit on it; to add one, another must come off. Or a whiteboard: when it is full, writing more means rubbing something out. The context window is the most tokens a single request can use. OpenAI’s documentation counts input, output and reasoning tokens against that limit. Your instructions, the conversation so far, any policy pages you supply and the reply share the space. The size differs between models, so this session states no real figure; read your model’s current value in its documentation. The figures in the picture below are invented.

 Invented window: 8,000 tokens (your model's real size differs)

 +---------------------------------------------+
 | Instructions ............................ 300 |
 | Conversation so far .................... 1,200 |
 | Policy page you supply ................... 600 |
 | Room left for the reply ................ 5,900 |
 +---------------------------------------------+
 Each extra line you send leaves less room for the reply.

Failure first, then cause. A support thread grows to 40 messages. Message 1 says an employee’s laptop will not start after an update; messages 2 to 40 are about a password reset and a printer. The summary talks about the password and the printer. The original complaint is lost. Why? Partly the limit, and partly how models read long text. In one study, performance was often highest when the relevant information sat at the start or end of the input, and degraded significantly when it sat in the middle of a long context.

The same shape appears in everyday life. A doctor who reads just the latest pages of a thick file misses the allergy noted at the first visit. In the Copilot, a long chat can drop the employee’s first message, the one that explained the problem. Bring the three right pages to the desk, not the whole filing cabinet. Choosing the right pages is the retrieval idea of module 4.

The reply also sits inside the window. If the prompt and history fill most of it, little room is left for the answer, and OpenAI’s documentation notes that tokens generated beyond the limit may be truncated, which shows up as a reply that stops mid-sentence. The request setting max_completion_tokens sets an upper bound on the tokens generated for a completion, including visible output and reasoning tokens.

A model does not remember between calls. Each text generation request is independent and stateless, and a multi-turn conversation works by sending the earlier messages again. That is why history takes up window space, and a later session returns to it.

How much does an LLM API call cost, and how do I estimate it before sending?

A taxi meter charges for distance travelled. A phone plan by data used charges for what you send and receive: one short message costs almost nothing, a thousand long ones add up. A model call is metered the same way. The cost of a call is its input tokens and its output tokens, each multiplied by that provider’s price. OpenAI’s pricing page quotes prices per 1M tokens and lists input, cached input and output as separate prices. Prices change, so this course does not state one as fact: the prices here are invented, and you look up your provider’s current price when you need a real one.

The formula for the input side is short. This is the line code/count_tokens.py uses, with sends the number of times you send the text and price_per_million a sample price you change to your provider’s:

cost = tokens * sends / 1_000_000 * price_per_million

A worked example with invented figures. Suppose the price is 2.00 per million input tokens and each text is sent 500 times:

Text Tokens (invented) Tokens over 500 sends Cost at 2.00 per million
A short question 20 10,000 0.02
Question plus one policy section 150 75,000 0.15
The whole policy file 2,400 1,200,000 2.40

For the last row: 2,400 × 500 = 1,200,000 tokens, which is 1.2 million, so the cost is 1.2 × 2.00 = 2.40. Sending the whole file costs about a hundred times what the short question does, on each send. That is the surprise behind the bill that jumps when someone pastes the whole policy file into each question.

An estimate is not the final figure. After each call, providers that report usage send figures back with the reply: the number of tokens in the prompt, the number in the generated completion, and the total used in the request.

code/read_usage.py prints the first two of its three figures like this (the third is a total of the two):

usage = response.usage
print("Tokens you sent:  ", usage.prompt_tokens)
print("Tokens it wrote:  ", usage.completion_tokens)

Your own count is an estimate, closest for the model whose tokeniser you used, so it may differ from “Tokens you sent”. The usage figures are what the provider counted. Input and output tokens are priced separately, so the total alone is not the cost: price each side, then add them. Estimate before you send; trust the usage figures after.

Two free websites, no install. Google AI Studio, where you got your key in session 1, shows a token count for your prompt, and the count grows as the chat grows. It shows tokens, not money, and where the count sits on the page has changed between versions. The site llm-prices.com is a price calculator: type the input tokens and the output tokens and it shows the cost for many models, with input and output priced separately. One counts, the other prices. Different models use different tokenizers, so a count from one model priced on another is approximate.

Three more programs: type your own, then watch the history grow. These three programs call a model and use the same three settings as code/read_usage.py. The prices inside them (0.30 per million input tokens, 2.50 per million output tokens) are invented sample prices; put your provider’s prices there.

  • code/ask_usage.py asks Your question: and prints the reply, the input, output and total tokens, and one cost line with input and output priced separately. Run it with python ask_usage.py.
  • code/two_questions_usage.py asks question 1, then question 2 in the same conversation. The model remembers nothing between calls, so send 2 carries question 1 and reply 1 again. It prints send 1 tokens, send 2 tokens, “of which question 1, sent again” (the same as send 1), “of which new (reply 1 + question 2)” (the difference), the tokens each reply wrote, and the input tokens paid over both sends. Run it with python two_questions_usage.py.
  • code/chat_usage_loop.py is a read loop: type a message at You: and type quit to stop. After each turn it prints sent this turn: S = C carried from earlier turns + N new (last reply + your message), the tokens written, and the running input and output totals and cost so far. Run it with python chat_usage_loop.py.

Hold a chat of five or six turns and watch C. In a chat, each turn sends the whole conversation again, so the input count grows each turn and you pay for the history again.

Three situations where tokens bite. Failure first, then cause:

  1. A support thread reaches 40 messages and the summary forgets the laptop that would not start, reported in message 1. The window is limited, and models use the middle of a long input less well than its start and end.
  2. The monthly bill is far above the estimate after the whole policy file was added to each question. The file’s tokens are paid for on each send, not once.
  3. A reply stops in the middle of a sentence. The prompt and history left too little room, and tokens beyond the limit may be truncated.

Each surprise comes from not counting, and each fix starts with a token count. Read each row of the table below as a lesson and the design answer it leads to.

What you learned The design answer Built in
Limits and prices are counted in tokens Count what you send before you send it This session
Prompt and reply share one window Send just the pages a question needs Module 4
Cost repeats on each send Estimate cost per question, not per project This session
Long chats push early messages out Trim or summarise history Module 3
A change that feels better may not be Judge each change against the test set Later sessions

How do I count tokens in Python with tiktoken?

Think of a tile cutter for text: you feed in a sentence and it hands back the tiles, one per token, so you can count them. tiktoken is such a cutter, a fast tokeniser for use with OpenAI’s models; install it with pip install tiktoken. It names its tokenisers encodings, and o200k_base is one of them; it can also return the tokeniser for a specific OpenAI model. On average, in practice, each token corresponds to about 4 bytes of text, according to its README. The count is an estimate for any other model, since another model may split the same text differently.

tokenizer = tiktoken.get_encoding("o200k_base")

for label, text in texts.items():
    tokens = len(tokenizer.encode(text))

This is from code/count_tokens.py. It does not call a model, so it needs no settings. To try it yourself: first guess, then check. The script counts two texts: a short question, and the same question with a long document pasted in front. Guess both counts first, run the script, and see which guess was furthest off. Then change the question text in the script to a question of your own and run it again.

Basic assignment. Run the counter on its two texts, then change price_per_million to your provider’s current price and run it again. Write down which text costs more, and by roughly how many times the question alone. Then run code/read_usage.py to see the real counts a model reports back for one question. The Tiktokenizer page (tiktokenizer.vercel.app) is an online playground for the same library: pick a gpt-4o model, paste a string and see the token count and how it is split. tiktoken maps gpt-4o to o200k_base, the encoding the script uses, so it is a good way to see why a text counted higher than you thought. Then run code/ask_usage.py with a short question and a long one, and code/chat_usage_loop.py for a few turns, and note how the “carried from earlier turns” figure grows. The project file gives the exact steps.

What is a test set, and how do I write one for an AI assistant?

A driving test uses the same route for everyone, so results can be compared. A pre-flight checklist has the same items each time, so a skipped one shows up. A test set works the same way: a fixed list of questions, each with the answer you expect, that you run again whenever the prompt or the model changes. Without one, “this prompt is better” is a feeling; with one, it is a count.

Project step: write the ten questions. In prompts/test_set.json, write ten entries of this shape, about leave, laptops, expenses and tickets:

{
  "id": 1,
  "question": "How many leave days can I carry over?",
  "expected": "Up to 5 unused days, used by 31 March (Leave Policy 4.2).",
  "needs_source": false
}

Include one question the sample policies do not answer, with the expected answer “I don’t know”. Write the expected answer before you see the model’s, otherwise you grade to fit it. Leave needs_source as it is: the next session marks which questions need a source document. The test set is run against the model in later sessions, so keep it.

Practice: say the answer. Answer the question in the Model answer section aloud in four beats before you read the version below. Then pick one of the three situations above and say which beat explains it.

Flashcards: LLM, token and context window questions with answers

Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.

1. What is a foundation model? It is one model trained on broad data at scale that can be adapted to a wide range of tasks. A large language model is one: the same model can answer, label or draft depending on the instructions, which is why one model can serve many jobs in the Copilot.
2. How did handling a question change from rules to foundation models? Rules needed a person to list the phrasings and broke on the unlisted ones. Machine learning learned from examples, but a new task usually meant training again on new labelled examples. Foundation models are trained broadly and steered by a prompt, at the cost of being wrong in many tasks at once.
3. What is a token, and why is a word not a token? A token is the small piece of text a model reads and writes: a word, part of a word or punctuation. A rare word can be several tokens. For English, about 4 characters or 0.75 words per token is a rough rule; the reliable figure comes from counting.
4. How does a model write a reply? One token at a time. It picks a likely next token given the tokens so far, adds it to the text and picks again until the reply is complete. Tokens you send and tokens it writes are both counted.
5. What are the five steps of building a chat reply? Pick the policy section; build the request from the role and tone, the section, the question and the settings; read the whole request as tokens, with attention linking the question to the section; write the reply token by token, each pick a draw; then give the answer, which can change with the settings. In this session you do the first step by hand.
6. What is attention, in one picture? For the word it is working on, the model scores the other words for how much they matter and mixes in more of the high scorers, like a reader highlighting what "it" refers to. A word can look back at any other word, so meaning is found wherever it sits.
7. What is a transformer, and how does it differ from attention? A transformer is a model built from many attention layers stacked one on another: in each layer the tokens look at the other tokens and update their meaning, and after many layers the last position predicts the next token. Attention is the single step; the transformer is the machine made of many such steps.
8. What shares the context window? Instructions, the conversation so far, any pages you supply and the reply itself, counting input, output and reasoning tokens. The more you send, the less room remains, and tokens beyond the limit may be truncated, so a reply can stop mid-sentence.
9. How is the cost of a call worked out? Input tokens and output tokens, each multiplied by the provider's price per million tokens. Estimate before sending by counting tokens, then read the prompt, completion and total figures that come back with the reply.
10. Why does pasting the whole policy file into each question get expensive? Its tokens are paid for on each send, not once. In the invented example the whole file cost about a hundred times the short question per send, and it also fills window space the reply needs. The same happens in a chat: each turn sends the whole conversation again, so the input count grows each turn.
11. What is a test set and why write the expected answers first? A fixed list of questions with the answers you expect, run again after each change so "better" is a count. Writing the expected answer first stops you from grading to fit whatever the model says.

Model answer: how does a model turn a question into a reply, and what will sending it cost?

The core question this session answers: “A colleague asks: how does a model turn my question into a reply, and what will sending it cost?”

Four beats, in this order: one broad model · tokens and attention · the window · the cost. Skip any of these four and the answer is incomplete.

  1. One broad model: a foundation model, trained on broad data, steered by your instructions.
  2. Tokens and attention: text is cut into tokens, attention links the words of the request, and the reply is built one token at a time.
  3. The window: instructions, history, supplied pages and the reply share one context window.
  4. The cost: input tokens plus output tokens, each at the provider’s price; estimate first, read the usage figures after.

The full answer, said aloud:

(One broad model) “A large language model is a foundation model: one model trained on broad data that a prompt can steer to answer, summarise or label, instead of a model retrained for each task.

(Tokens and attention) Your question is cut into tokens, and as a transformer the model lets each word look back at the other words and weigh the ones that matter, so meaning is found wherever it sits. It then picks a likely next token again and again to build the reply.

(The window) Your instructions, history, any pages and the reply share one context window.

(The cost) A call costs its input tokens and output tokens at the provider’s price per million, so I count tokens first to estimate, and check the usage figures that come back.“

Missing a beat is a failure state. If you stop before the cost, the follow-up is what a thousand of these calls would cost. Practise until the four come out without prompting.

Why this matters: where do you use tokens, context windows and cost estimates?

In a budget conversation: before a pilot, someone asks what it will cost for a typical set of questions. A token count times a price, multiplied by the expected number of questions, is an answer you can defend, and the usage figures later show how close you were.

In a design review: deciding what the Copilot sends with each question is a token decision. Knowing the window is shared tells you why history has to be trimmed and why you send the right pages, not the whole file.

When reading about models: documentation, pricing pages and release notes are written in tokens and windows. A reader who knows what a transformer’s attention does and why a token is not a word can follow them.

After this unit: what is the assignment and what comes next?

Assignment. Count the tokens for the two texts in the script, and for texts of your own, and estimate the cost of answering each, then write the ten test questions with expected answers in prompts/test_set.json. The brief itself is the project file.

What’s next. The next session asks why the same question can get different answers: how the next token is chosen, what temperature changes, why a model makes things up, and other limits. You also mark which test questions need a source document.

Want help building this with someone?

Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.

Ask about mentor-led training

TechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.