Courses / Agentic AI from Scratch (Python)
Session 3 of 40
Temperature, sampling, hallucinations and other limits
Notes
Why Does an LLM Give Different Answers? Temperature, Sampling, Hallucinations and Other Limits Explained
Agentic AI Foundations · Session 3 — Temperature, sampling, hallucinations and other limits
Before you start: what do you need to experiment with temperature and hallucinations?
You need the setup from session 1: Python, the course code, and a model endpoint that accepts the OpenAI request format. Your three settings (LLM_BASE_URL, LLM_API_KEY and LLM_MODEL) sit between the quotes at the top of each script, or you can set them in your terminal; if a setting is in the terminal, the terminal wins. No new library is needed. Keep your project folder open, because you will edit the test set you wrote in session 2, prompts/test_set.json. Have paper or a notes app ready for the Flashcards and the Model answer.
Summary: why can the same question get different answers from an LLM?
A model chooses each next token by a weighted draw from the likely candidates, and the temperature setting controls how adventurous that draw is, which is why the same question can get different answers. Temperature changes how much a reply varies, not how much the model knows, so a low setting can repeat a wrong answer just as steadily as a right one. A hallucination is a fluent, confident reply that is invented, such as a policy section that does not exist, and it is most likely when the model is asked for a fact it was not given. Other limits are a training cut-off, no live data, shaky arithmetic and weaker use of long inputs, and each one has a design answer in the Copilot. The project step marks which of your test questions need a source document.
Key concepts: sampling, temperature, top-p and hallucination defined
| Term | Definition |
|---|---|
| Sampling | How a model picks its next token: it scores the possible next tokens and draws one from the likely ones instead of just taking the top choice. |
| Temperature | A setting for how random the choice of each next token is. Lower values make output more focused; higher values make it more varied. |
| Top-p | A second sampling setting, also called nucleus sampling, that limits the pick to the likeliest tokens whose probabilities add up to a chosen share, such as the top 10 percent. |
| Hallucination | A reply that is fluent and confident but invented or wrong, such as a figure or a policy section that does not exist. |
| Grounding | Giving the model the source text to answer from, so the answer can be checked against it. |
| Training cut-off | The date up to which a model’s training text was gathered; events after it are not in what the model learned. |
| Live data | Facts that change after training, such as today’s open tickets. A plain model call does not see them unless your code passes them in. |
needs_source |
A true or false mark on a test question saying its answer must come from a policy or ticket, not general knowledge. |
Explanations: how do sampling, temperature, hallucinations and model limits work?
Where are we? What does this session add to the HelpDesk Copilot?
Session 1 made a first call and session 2 counted what a call costs. This session studies how the model behaves when you call it, and adds three things to the project. A script chapters/03-temperature/temperature_lab.py runs one question at three temperatures. A script chapters/03-temperature/policy_answer.py sends the same question with a policy section and runs it at two temperatures. Your test set gains a mark on each question saying whether it needs a source document. The new parts sit beside what you built earlier: the first call (session 1), the token counter and the test set of questions (session 2). The terminal chat, the fixed answer format and the main prompt come in sessions 4 to 6, and policy search in module 4.
| Box | What it does for the employee |
|---|---|
| Temperature lab | Shows how the same question behaves at three settings |
| Policy answer demo | Shows the same question answered from a pasted policy section |
| Test set marked for sources | Tells you which questions may be answered from general knowledge and which need a policy page |
How does the Copilot answer a question, and where do sampling and temperature fit?
Think of a help desk clerk answering a phone call. The clerk finds the right page of the handbook, reads the page with the caller’s question in mind, then speaks the answer one word after another. The Copilot follows five steps like these, drawn here for the carry-over question:
Today’s focus is step 4, inside the model. The same five steps as text:
Employee: "How many leave days can I carry over?"
|
v
1. Pick the policy section leave policy, 4.2 "Carry-over"
|
v
2. Build the request role and tone + section + question
settings: model, temperature
|
v
3. Read it all as tokens attention links the question to the section
|
v
4. Write the reply, token by token each pick is a draw (temperature)
|
v
5. Answer "Up to 5 days, used by 31 March" or "Not in the policy: raise a ticket"
Reading it in plain words: steps 1 to 3 decide what the model sees, step 4 is where the words are chosen one at a time, and step 5 is the answer the employee reads. This session is about step 4. Sampling is how each pick is made, and temperature, a setting you send in step 2, decides how adventurous the draw is. The same picture explains the made-up answer: if step 1 hands the model no section, step 4 still writes something fluent. Two demos later in this page fill steps of the picture: the temperature lab changes only step 4’s setting, and the policy answer demo fills steps 1, 2 and 4.
How does an LLM pick the next word? A quick refresher on attention and the transformer
Before the draw, a quick look back at session 2. Picture a bus timetable clerk finishing a sentence on a sign: “Up to 5 unused days can be carried over to next …”. The clerk reads the words already written, works out what they mean together, and then thinks of the word that most likely comes next. A model does the same in two parts. Attention lets each word look at the other words and score how much they matter. The transformer stacks many attention layers, so the meaning builds up layer by layer. At the end, the last position holds the meaning so far and gives odds for the next word. This picture, from session 2, shows one full pick and the loop back to the start:
Now the same loop on a Copilot sentence. The odds are illustrative, written for these notes:
"Up to 5 unused days can be carried over to next ..."
1 Attention "carried over" links to "days"
2 Layers build the meaning: a period comes next
3 Odds year 70%, month 20%, week 6%
(illustrative)
4 Pick one is drawn: "year"
5 Add "... to next year"; go round again
Step 3 gives odds, not a single answer: more than one word is possible. A second way to see it is your phone’s autocomplete showing three suggestions. Which one gets picked? That choice is sampling, and the next sections explain it.
What is temperature in an LLM, and how does sampling choose the next token?
Start with a puzzle: the same question can get different answers. Ask the Copilot “Write a one-line welcome for a new joiner” three times and you might get three different lines, such as “Welcome aboard, we are glad you are here”, “So happy to have you on the team!” and “A warm welcome: we are excited to start with you”. These lines are fine, and the variety is good. Now ask “How many days of sick leave do I get?” twice and you might get “ten days” and then “twelve days”. Here the variety is a problem, because just one figure can be right. These replies are illustrative, written for these notes. Where does the difference come from?
It comes from how each next word is chosen. Suppose the reply so far is “Welcome to the team, we are so …” and the model’s candidates, with invented odds, are glad (40%), happy (30%), excited (20%) and thrilled (10%). Which should it take? Option 1 is to take the top one each time: the same reply on each run, flat and prone to repeating itself. Option 2 is to draw by the odds: natural and varied, but sometimes an unlikely word. Sampling is how the model makes this pick, and temperature is the dial for how adventurous the draw is.
Picture a raffle in which each possible next word holds some tickets. Likely words hold many, unlikely words hold a few, and the model draws one ticket. Run the raffle again and a different ticket can win. That draw is sampling: the model scores the possible next tokens, then picks one of the likely ones instead of just taking the single top choice. A second picture: autocomplete on your phone, where you sometimes tap the second suggestion. A reply is built from a long run of such picks.
Failure first. An employee asks “How many days of sick leave do I get?” and is told “ten days of paid sick leave a year”. A colleague asks the identical question and is told “twelve days, paid in full”. Both replies sound sure. The cause is not a bug. The two runs drew different tokens, and nothing in either run was checked against a policy.
Here is the draw as a picture. The odds are invented for illustration:
Next token after "Up to 5 unused annual-leave ..."
(illustrative odds, invented for this page)
LOW temperature HIGH temperature
days ################ 90% days ######## 45%
leave ## 7% leave ##### 28%
hours # 2% hours ### 17%
bananas . 1% bananas ## 10%
Temperature is the setting that reshapes those odds. Think of a chef following a recipe versus improvising: at a low temperature the chef repeats the recipe, and at a high one tries new combinations. Or think of rolling a loaded die. The likely tokens come up more often, and temperature decides how loaded the die is. OpenAI’s Chat Completions reference describes temperature as a number between 0 and 2, where higher values such as 0.8 make the output more random and lower values such as 0.2 make it more focused and deterministic.
Top-p is a second dial. It is an alternative to temperature, called nucleus sampling: the model considers just the tokens that make up the top share of probability, so a top-p of 0.1 means the top 10 percent. The reference gives it a range of 0 to 1. The same reference generally recommends changing temperature or top-p but not both, so change one setting at a time when you compare. Providers also treat settings in their own ways. One hosted provider that mimics the OpenAI API says a temperature of 0 is converted to a tiny positive number, so check your own provider’s documentation if a setting seems to do nothing.
| Setting | What it changes | Range in OpenAI’s reference |
|---|---|---|
temperature |
How random each pick is | 0 to 2 |
top_p |
How much of the top probability the pick is limited to | 0 to 1 |
| Neither sent | The provider’s own default applies | Not applicable |
In the code, temperature is one line inside the call that asks the model. This is the call from chapters/03-temperature/temperature_lab.py; the reply text is response.choices[0].message.content:
response = client.chat.completions.create(
model=os.environ.get("LLM_MODEL", LLM_MODEL),
messages=[{"role": "user", "content": question}],
temperature=temperature,
)
When should you turn temperature down, and when does variety help? Use a low setting when the next step is checking or sorting: a policy answer, where steady wording is easier to check, or a ticket category, where the same ticket should get the same label. Use a higher setting where being different is the point: three friendlier rewordings of an announcement, or a list of possible titles for a help page. If you want an earlier reply back word for word, a low setting makes it likely, not certain. OpenAI describes even its seed option as a best effort, and says determinism is not guaranteed.
Temperature changes how much a reply varies, not how much the model knows. A low setting makes a wrong answer repeat steadily; it does not make it right.
How does an LLM build a whole sentence? Attention, layers, temperature and sampling together
Think of a relay team passing a baton, but the course is a sentence, and the team runs the same lap over and over. Each lap adds one token. Now that you have seen sampling and temperature, here are all the parts in one picture, in the order they run:
One line for each part:
| Part | What it does |
|---|---|
| Attention | Each word looks at the other words and scores how much they matter. |
| Transformer layers | Many attention layers in a row build up the meaning of the text so far. |
| Odds for the next token | The last position turns that meaning into a list of possible next tokens with odds. |
| Temperature | Reshapes those odds: a low setting sharpens them, a high one flattens them. |
| Sampling | Draws one token from the reshaped odds. |
| Token added | The new token joins the reply, and the loop starts again until the reply is complete. |
The whole reply is this lap repeated. That is why the same question can come out differently: the draw happens on each lap.
What is an LLM hallucination, and how do you reduce it?
Think of a new hire who will not say “I don’t know” and makes up a plausible answer instead of checking. Or a map with an invented road: it looks complete until you drive there. A third picture is a student guessing on an exam. A guess can score and a blank cannot, so guessing wins. A hallucination is such a reply: fluent and confident but invented or wrong.
Failure first. Suppose the Copilot is asked about sick leave and replies:
Under Leave Policy 4.9 you get twelve days of sick leave a year, paid in full.
This reply is illustrative, written for these notes. It has a precise figure, a section number that looks real and an even tone. There is no section 4.9 in the leave policy, and the sample policies give no sick-leave figure. Nothing in the wording shows doubt.
Why does the model write this? In everyday words, think of a tour guide who has not visited the town and still names the old bridge and the best cafe in a calm voice. A model writes what a good answer would look like, and it was not given the real text. One research account is that language models hallucinate because training and evaluation procedures reward guessing over acknowledging uncertainty, much as an exam rewards a guess over a blank. A confident tone tells you about the writing, not about the facts.
The contrast between an answer with and without the source is the heart of the matter:
WITHOUT THE POLICY TEXT WITH THE POLICY TEXT
Question Question + text of Leave Policy 4.2
| |
v v
Model asks itself: Model asks itself:
"What would a policy "What does THIS text
plausibly say?" say?"
| |
v v
Fluent answer, maybe invented Answer you can check against 4.2
Give the model the source and it has something to quote and you have something to check. No single defence is enough, so layer them:
| Defence | What it does | What it does not fix |
|---|---|---|
| Supply the policy text in the prompt | Gives the model words to answer from | It may still misquote them |
| Say what to do when the text is silent | Gives it permission to say “I don’t know” | It may ignore the rule, so test it |
| Ask for the section number | Gives you something to check | A section number can be invented too |
| A low temperature | Makes wording steadier | It does not make a wrong answer right |
| A person checks the figure | Catches what slipped through | It takes time, so use it where a mistake costs most |
How to catch a hallucination by hand: find the figure or the citation in the reply, open the real document, and compare. In the example above, you would open the leave policy, look for section 4.9, and find that it is not there.
What causes an LLM hallucination? A diagram and a real-time example
First the failure. Someone asks a chat assistant: “When is the last bus on route 12 from the Riverside stop to the airport?” and gets back: “The last bus leaves at 11:42 pm.” This reply is illustrative, written for these notes, and the route and time are invented. It sounds exact and sure. Nobody gave the assistant a timetable, so the time is made up. A local clinic’s opening hours would fail the same way.
Why does it happen? Think of a tour guide who has not visited the town and still names the old bridge in a calm voice, or a student guessing on an exam: a guess can score and a blank cannot. Here is the same idea in the model’s own terms. Ask “How many sick days do I get?” with no policy text, and the odds after “You get ___ days” spread over several plausible numbers, because many companies’ policies seen in training give different figures. The draw picks one, and the reply reads as confident. With the policy section supplied, the odds concentrate on the section’s figure. The odds are illustrative:
Three causes, in plain words:
- It predicts plausible next words; it does not look facts up. A word that fits is not the same as a word that is true.
- Nothing in the loop checks truth. Attention, layers, odds and the draw have no step that asks whether the sentence is so.
- No source was supplied. With nothing to read, the model fills the gap with what a policy or a timetable would usually say.
How to catch it: find the exact figure, open the real timetable, clinic page or policy, and compare. How to prevent it: supply the document and say what to do when it is silent.
What happens when one question hits four LLM limits at once?
A hallucination is one limit of a plain model. This session also covers four more, and a single ordinary question can hit several of them. Picture an employee writing to the Copilot:
“The leave policy changed last month. After the 3 days I take next week, how many days do I have left, is my laptop ticket still open, and can you summarise the 60-message thread?”
Each part of that question meets a different limit:
- “The leave policy changed last month” meets the training cut-off: the change is not in what the model learned.
- “Is my laptop ticket still open” meets no live data: a plain call cannot see the ticket system.
- “After the 3 days I take next week, how many days do I have left” meets arithmetic: a sum worked from memory can be off by a day.
- “Summarise the 60-message thread” meets long documents: details in the middle or at the start of a long input can be lost.
The sections below take each limit in turn and give the design answer. Keep this one question in mind: it is the table’s “what it looks like” column.
What are the other limits of an LLM: training cut-off, no live data, arithmetic and long documents?
A hallucination is one of several ways a model lets you down. Four more limits matter for the Copilot.
The training cut-off. Think of a printed encyclopedia: accurate for the year it was printed, silent about last month. Or a colleague back from a long leave, who knows the office as it was and does not know about the new policy. Models are often published with a claimed cut-off date, the date up to which training data was gathered, and such a date matters wherever the model must give up-to-date information. Failure first: asked about a recent change, the model describes the old rule in a confident voice. The cause is that the change is not in what it learned.
No live data. A plain model call has no link to your ticket queue. Ask “How many tickets are open today?” and it may produce a plausible number that nobody looked up. Your program has to fetch the data and hand it over in the prompt. Some providers also offer a tool for this: OpenAI’s web search tool lets models search the web for the latest information before answering. Later in the course the Copilot gets tools for its ticket data.
Arithmetic. Research reports that transformer language models often struggle with seemingly easy tasks like arithmetic. Failure first: asked for the leave left after a carry-over and a booking, the Copilot gives a total that is off by a day. Think of a clerk who writes sums from memory instead of using a calculator: most come out right and some are quietly wrong. The safe habit is to work the sum out in code and let the model explain the result.
Long documents. The context window is the maximum number of tokens in a single request, and it includes input, output and reasoning tokens. Beyond the limit, there is also a quality problem. In one study of long inputs, performance was often highest when the relevant information sat at the beginning or end of the input, and degraded significantly when it sat in the middle. A doctor who reads just the latest pages of a thick file misses the allergy noted at the first visit, and a model summarising a 40-message thread can miss the complaint in message 1. Send short, relevant pages instead of the whole file.
| Limit | What it looks like | What the Copilot does about it |
|---|---|---|
| Training cut-off | “The policy changed last month”: the old rule, stated confidently | Supply the current document |
| No live data | “Is my laptop ticket still open?”: a status nobody looked up | Fetch the data in code and hand it over |
| Arithmetic | “After the 3 days, how many are left?”: a total off by a day | Work the sum out in code |
| Long documents | “Summarise the 60-message thread”: a summary that misses message 1 | Send short, relevant pages |
How do I run one question at three temperatures and catch a made-up answer?
Hands-on lab and basic assignment. In your unit folder, run python temperature_lab.py (the file is chapters/03-temperature/temperature_lab.py). The script sends one question three times, at temperature 0.0, 0.5 and 1.0, and prints each reply. The question asks how many days of sick leave employees at Fernhill Systems get each year. Fernhill Systems is a made-up company and no text is supplied, so any figure in a reply is made up. The core of the script is this loop:
question = "How many days of sick leave do employees at Fernhill Systems get each year?"
for temperature in [0.0, 0.5, 1.0]:
response = client.chat.completions.create(
model=os.environ.get("LLM_MODEL", LLM_MODEL),
messages=[{"role": "user", "content": question}],
temperature=temperature,
)
print(f"\n--- temperature {temperature}")
print(response.choices[0].message.content)
Read the three replies slowly. Which wording changed, and did any figure change? Run the script a second time: at 0.0 the replies come out nearly the same, and at 1.0 the wording changes more. Then circle each figure in the three replies and mark the ones nobody supplied: those are your made-up answers. Try it with illustrative replies first, written for these notes:
- Temperature 0.0: Employees at Fernhill Systems get ten days of paid sick leave a year.
- Temperature 0.5: Fernhill staff get ten days of paid sick leave each year, and more with a doctor’s note.
- Temperature 1.0: Staff receive twelve days, plus three extra days in the first year.
The figures ten, twelve and three were supplied by nobody, so each is made up, even though the first two replies agree. Agreement is not evidence, and a low temperature does not stop an invented figure; it only makes it repeat. Finally, change question to another question that no supplied text answers, run it again, and write down which figures are made up.
How do I make the model answer from a policy section instead of guessing?
Now send the same question with the policy text. In chapters/03-temperature/policy_answer.py you paste section 5 of the leave policy into a string, add a system message that sets the role and tone, and run the call at temperature 0.0 and 1.0. The system message says: answer only from the policy section given, and if the section does not answer, say so and suggest raising a ticket. Run it with python policy_answer.py. The parts that matter:
messages = [
{"role": "system", "content": system_message},
{"role": "user", "content": "Policy section:\n" + policy_section + "\nQuestion: " + question},
]
for temperature in [0.0, 1.0]:
response = client.chat.completions.create(
model=os.environ.get("LLM_MODEL", LLM_MODEL),
messages=messages,
temperature=temperature,
)
In terms of the five-step picture, you do step 1 by hand (you chose section 5 and pasted it), the messages list is step 2 (role and tone, section and question) and the loop is step 4. Section 5 tells an employee how to report sick leave, but it gives no number of days. A grounded reply therefore says the policy does not say, and suggests raising a ticket. Look for two things: the wording may differ between 0.0 and 1.0, but the fact should hold, and no invented figure should appear. If a figure does appear, treat it as a made-up answer and circle it. An illustrative grounded reply, written for these notes: “The sick leave section does not give a number of days. Please raise a ticket with HR.”
Project step. Open prompts/test_set.json and set needs_source to true or false on each of your test questions. A question needs a source when its answer must come from a policy or a ticket and cannot be general knowledge. Examples: “How many days can I carry over?” needs a source, because the answer sits in Leave Policy 4.2. “What is the status of ticket T-1042?” needs a source, because the ticket data holds the answer. “What does Wi-Fi stand for?” does not, since it is general knowledge. “Suggest a polite subject line for a reminder email” does not either, since it asks for wording, not a fact. Stop when each entry is marked, with at least three true and two false.
What do these limits mean for the design of the Copilot?
Three situations from the Copilot, failure first, then cause. The replies are illustrative, written for these notes.
- Two different answers. Two employees ask the same sick-leave question and get different day counts. The model drew different likely tokens, and neither answer was checked against a policy.
- The invented section. The Copilot cites “Leave Policy 4.9” and no such section exists. No policy text was supplied, so the model wrote what a policy would plausibly say.
- The confident number. An employee asks how many tickets are open today and gets a specific figure. The model has no live view of the ticket system, so it produced a plausible one.
What this means for your design:
| What you learned | The design decision | Where it shows |
|---|---|---|
| Replies vary from run to run | Low temperature for policy answers; compare one setting at a time | temperature=temperature in the call |
| A sure tone hides an invented fact | Answer from supplied text, name the section, say so when it is not there | The main prompt, written later |
| The model has no live data | Fetch ticket facts in code and pass them in | Ticket tools, built later |
| Arithmetic can slip | Calculate in code | Tools, built later |
| Some questions need a document | Mark them, so each is tested against the right source | needs_source in the test set |
OpenAI recommends writing tests that measure how a prompt behaves, so you can watch performance as you edit prompts or upgrade the model. That is why the marked test set matters: it lets you check, change by change, whether the invented answers went away.
Practice: explain and sort. First, answer the Model answer question below aloud in four beats before reading it. Second, for each of these four situations, decide whether the cause is the draw, a missing source, no live data or arithmetic, and write your reason in one sentence:
- Two runs of the same question give different day counts.
- A reply quotes a section that is not in any file you supplied.
- A reply gives a number for today’s open tickets.
- A reply gives a wrong total for leave left.
Flashcards: temperature and hallucination questions with answers
Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.
1. What is sampling, and why can one question get different answers?
Sampling is how a model picks its next token: it scores the possible tokens and draws one from the likely ones. Because the draw is weighted, not fixed, two runs of the same question can pick different tokens, and the wording or a figure can change.2. What does temperature change, and what does it leave alone?
It changes how random each next-token pick is: a low value gives focused, steadier wording and a high value gives more variety. It leaves the model's knowledge alone, so a low setting can repeat a wrong answer just as steadily.3. What is top-p, and how should you use it with temperature?
Top-p limits the pick to the likeliest tokens whose probabilities add up to a chosen share, such as the top 10 percent. OpenAI generally recommends changing temperature or top-p but not both, so you can see which setting caused a difference.4. When do you turn temperature down, and when does variety help?
Turn it down for policy answers and ticket labels, where steady wording is easier to check. Allow more for tasks like rewording an announcement three ways or brainstorming titles, where being different is the point.5. What is a hallucination, and what is a likely cause?
A hallucination is a fluent, confident reply that is invented or wrong, such as a section number that does not exist. One research account is that training and evaluation reward guessing over admitting uncertainty, so the model writes something plausible.6. Why does a made-up answer sound so sure?
The model writes what a good answer would look like, so the tone comes from the style of the text, not from checking facts. A confident voice tells you about the writing, not about whether the figure is real.7. How do you reduce hallucinations in the Copilot?
Layer defences: supply the policy text, say what to do when the text is silent, ask for the section number, lower the temperature and have a person check high-stakes figures. No defence in the list is enough alone, so measure the result with the test set.8. What is a training cut-off, and what is the related limit of no live data?
The cut-off is the date up to which a model's training text was gathered, so later changes are not in what it learned. Separately, a plain call does not see live facts such as today's tickets, so your code has to fetch them and pass them in.9. Why should arithmetic be done in code?
Research reports that language models often struggle with seemingly easy arithmetic, so a leave total can be off by a day. Working the sum in code and letting the model explain the result gives a figure you can trust.10. Why can a long chat lose the first message?
The context window is limited, and in one study of long inputs models often used information at the start and end better than information in the middle. A summary of a long thread can therefore miss the complaint that came first.11. What does the needs_source mark on a test question mean?
It says the answer must come from a policy or a ticket and cannot be general knowledge. It tells you which questions to test against a source document, and which the model may answer from what it already knows.12. Which step of the Copilot's five-step answer does temperature affect?
Temperature is a setting you send when you build the request in step 2, and it works in step 4, where the reply is written token by token and each pick is a draw. Steps 1 and 3, choosing the policy section and reading it, are not changed by it.Model answer: why did the Copilot give two different answers, and what do we change?
The core question this session answers: “A teammate asks: the Copilot gave two different sick-leave answers to the same question, and neither matches any policy. Why did that happen, and what do we change?”
Four beats a teammate listens for, in this order: why replies differ · why a figure was invented · what to change · how to know it worked. Skip any of these four and the answer is incomplete.
- Why replies differ: the model draws each next token from likely ones, and temperature sets how adventurous the draw is.
- Why a figure was invented: no policy text was supplied, and a confident tone is not evidence.
- What to change: supply the text, say what to do when it is silent, lower the temperature, and pass in live or calculated facts from code.
- How to know it worked: run the test set before and after, with the source-needing questions marked.
The full answer, said aloud:
(Why replies differ) “A model writes one token at a time, drawing from the likely ones. Temperature sets how adventurous that draw is, so the same question can give different wording and, at a higher setting, different figures.
(Why a figure was invented) The Copilot had no policy text for that question, and a model writes what is plausible, not what is checked. A sure tone proves nothing, and neither does two replies that agree.
(What to change) I would supply the relevant policy text, tell the model to say so when the text is silent, ask for the section number so I can check it, and use a low temperature. A lower temperature makes wording steadier, not truer. Facts that change, like open tickets, or sums, like leave left, I would have the code fetch or calculate and pass in.
(How to know it worked) I would run the test set before and after the change, with the questions that need a source marked, and compare the scores, so I am not relying on a feeling.“
Missing a beat is a failure state. The follow-up question lands where you went quiet: finish without the last beat and the next question is how you know it is better. Practise until the four come out without prompting, then stop talking and let the questions come.
Why this matters: where do temperature and hallucination checks matter at work?
In a conversation with a teammate or manager: people who build with models are asked why an answer changed and why it was wrong. The four beats give you a structure with no obvious gap, and the marked test set gives you evidence instead of an opinion.
On the job: the same ideas decide where a help desk bot is safe to use. Settings are chosen per task, figures are checked against a document before they reach an employee, and questions about live or calculated facts are routed to code. Teams that skip these checks learn about invented answers from their users.
In the rest of this course: session 4 turns this single call into a conversation; session 6 writes the main prompt that holds the rules from this session; sessions 13 to 19 add search over the policies, which fixes the invented-section problem at its source; and the testing sessions score your marked test set automatically.
After this unit: what is the assignment and what comes next?
Assignment. Ask one question at three temperatures, circle each figure in the replies and mark the made-up ones. Then change the question to another that no policy answers and repeat. The brief for the project work, marking which test questions need a source document, is the project file. Keep your marked test set: the next session builds the terminal chatbot you will run it through.
What’s next. The next session covers the Chat Completions API in depth: messages and roles, why the whole history is sent each time, streaming a reply as it is written, the model landscape and how to choose a model, the same call in the Responses API, and your first chatbot in the terminal.
Want help building this with someone?
Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.
Ask about mentor-led trainingTechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.