Courses / Agentic AI from Scratch (Python)
Session 7 of 40
Prompt engineering 2: step by step, Tree and Graph of Thoughts, a first look at ReAct, and chaining
Notes
Prompt Engineering Techniques: Step-by-Step Reasoning, Tree of Thoughts, Graph of Thoughts, ReAct and Prompt Chaining
Agentic AI Foundations · Session 7 — Prompt engineering 2: step by step, Tree and Graph of Thoughts, a first look at ReAct, and chaining
Before you start: what do you need to try step-by-step, tree and chained prompts?
You need the code folder for this session, chapters/07-reasoning-and-chains/, open in your terminal. It holds helpdesk_agent.py, the two prompts the model reads in model_prompts/ (summarise.md and system.md), and the data/ folder with the policies and tickets.csv. The program is the help desk helper from session 5, now as a chain of two calls. Set your three settings (LLM_BASE_URL, LLM_API_KEY, LLM_MODEL) as in session 1, either on the top lines of the script or in the terminal. No new library is needed. Have paper or a notes app ready for the Flashcards and the Model answer.
Summary: how do you make a prompt reason before it answers?
Session 5 gave the HelpDesk Copilot a helper that reads policy files and tickets by asking for them through tools, and session 6 gave it a better prompt. A reply that sounds confident has still not shown its working. This session adds four techniques. You ask the model to write its working before the answer on questions with several steps, you learn the idea behind Tree of Thoughts and Graph of Thoughts, where the model explores several lines of reasoning instead of one, and you preview ReAct, the pattern in which a model reasons, acts and looks at the result. Last, you split the helper’s job into a chain of two calls, summarise the ticket and then reply, and learn why a chain can lose information between its calls.
Key concepts: step-by-step reasoning, Tree of Thoughts, Graph of Thoughts, ReAct and prompt chaining defined
| Term | Definition |
|---|---|
| Step-by-step reasoning | Getting the model to write its intermediate steps before the final answer. Also called chain of thought. |
| Tree of Thoughts | A way of prompting in which the model explores several lines of reasoning, judges them and keeps the best, instead of following a single line. |
| Graph of Thoughts | An extension in which the model’s thoughts are the points of a graph and the links show which depend on which, so thoughts can be combined. |
| ReAct | A pattern in which a model writes a reasoning step, takes an action such as calling a tool, reads the result and reasons again. |
| Prompt chaining | Splitting a job into steps, each its own model call, where one call’s output becomes the next call’s input. |
| Chain | The set of linked calls in a prompt chain. In the helper, call 1 summarises the ticket and call 2 answers the question. |
| Thought | One unit of reasoning text the model writes as a step toward an answer. |
tool_choice |
A setting on a call that says whether the model must use a tool ("required") or may decide for itself ("auto"). |
Explanations: how do step-by-step, tree, ReAct and chained prompts work?
Where are we? What does this session add to the HelpDesk Copilot?
A reply from the Copilot follows five steps: pick the policy section, build the request, read it as tokens, write the reply token by token, and give the answer. Since session 5 the model does the first step itself, by asking for a file through a tool call. This session changes two things around that. The work is split into two calls: call 1 summarises the ticket, call 2 answers from the summary and the policy. And the model shows its working before the final answer. The program is chapters/07-reasoning-and-chains/helpdesk_agent.py, and the two prompts it reads are in model_prompts/.
What is step-by-step reasoning (chain of thought), and when should I ask for the working?
An employee writes: “I have 8 unused leave days. How many do I keep after 31 March?” Leave Policy 4.2 says up to 5 unused annual-leave days carry over, to be used by 31 March. A prompt that just says “answer the question” can return a bare “eight”, a wrong total with nothing to check it against. Nothing in the reply shows where the slip happened.
Now ask differently: work through the question in order, then answer. This is the “Think step by step” part of model_prompts/system.md, exactly as the file has it:
## Think step by step
Before you answer, work through it in order: what the question asks, which policy section covers
it, and what the section says about the figure or rule. Put that short working at the start of
the answer field, then the final answer.
A reply can then read: what the question asks, section 4.2 says up to five days carry over, the employee has eight, so the answer is five. If the total were wrong, the line where it went wrong would be on the page. This habit is called step-by-step reasoning, also called chain of thought: getting the model to write its intermediate steps before the final answer.
Two pictures help. It is like showing your working in a maths exam: the steps expose the slip that a bare answer hides. It is also like a pre-flight checklist: each item is read out, not assumed. A research paper on chain-of-thought prompting described a chain of thought as a series of intermediate reasoning steps and reported improved results on reasoning tasks for large models. A second paper found that adding the words “Let’s think step by step” before an answer made models better at such tasks than answering directly.
Steps are a tool for some prompts, not a habit to add everywhere. A plain lookup such as “who is ticket T-1042 assigned to?” has nothing to work out, so the extra lines just add length. OpenAI’s guidance for its reasoning models says that, because those models reason internally, asking them to “think step by step” or to “explain your reasoning” is unnecessary; check the guidance for the model you use before adding it. Because the helper’s reply is JSON, the working goes at the start of the answer field and the employee reads the final sentence of it.
What are Tree of Thoughts and Graph of Thoughts, and when are they worth the cost?
Picture a chess player. A beginner plays the first move that looks good. A stronger player considers three moves, imagines how each goes, drops the bad ones and plays the best. Or picture a planner with a map who sketches three routes before choosing. Step-by-step reasoning is the beginner’s one line of thought written out. Tree of Thoughts is the stronger player’s habit.
Failure first. Three tickets arrive on the day before a demo: a laptop that cannot join the office Wi-Fi, a request for a new monitor, and a password reset. One engineer. The model is asked which to do first, writes a tidy chain of steps and picks the monitor. The steps look reasonable, and the first choice was not compared with any other. A single line of reasoning commits early and does not look back.
The research paper that introduced Tree of Thoughts describes a framework that enables exploration over coherent units of text, called thoughts, that serve as intermediate steps toward solving a problem. It lets the model make deliberate decisions by considering several reasoning paths and evaluating its own choices, with looking ahead or backtracking when needed to make global choices. In plain words: propose some options, check each one, keep the promising ones, and carry on from there.
Problem
|-- Option A --> check --> dead end
|-- Option B --> check --> promising --> next step
'-- Option C --> check --> promising --> next step
(a graph also lets B and C be merged into one answer)
Graph of Thoughts is a later idea that goes one step further. Its paper describes modelling the information a model generates as a graph, where each unit of information, an “LLM thought”, is a point and the links show how one depends on another. That lets a prompt combine thoughts into one outcome or improve a thought with feedback. Think of a group of colleagues who each draft a section: the tree keeps the best draft, the graph lets you take a paragraph from each and merge them.
These are ideas about how to organise a model’s reasoning. They are not libraries you install in this course, and you can try the tree idea with one prompt: ask for three options, ask what goes wrong with each, keep the best, then answer. Here is the cost in a table:
| Technique | What the model does | Calls and tokens | Worth it when |
|---|---|---|---|
| Plain answer | Replies directly | One short call | A lookup, such as the assignee of T-1042 |
| Step by step | Writes its working, then the answer | One longer call | A sum or a rule applied to numbers |
| Tree of options | Proposes options, checks each, keeps the best | One long call, or several calls | A choice with several routes where a wrong answer is costly |
| Graph of thoughts | Also combines thoughts from different branches | Several calls | Rare for a help desk; know the idea |
Try the options version where a wrong answer is costly, and count the extra tokens as in session 2.
Basic assignment: take the puzzle of three tickets and one engineer with a demo tomorrow, and send it to a model twice in any chat tool you have: once with a step-by-step prompt (list the facts, apply the rule, give “Answer:”), and once with a tree-of-options prompt (propose three orders, say what goes wrong with each, keep the best, then answer). Compare the two replies: did the options version find a reason the first choice was weak? Write down which one you would put in the Copilot, and why.
What is ReAct, and how is it different from step-by-step reasoning?
Picture a detective. A detective who just thinks can build a story from what they already know and be wrong. A detective who forms a hunch, checks a record, sees what it says and then revises the hunch gets closer to the truth. ReAct gives a model that second habit. The paper that introduced it has the model write its reasoning and its actions in alternation. The reasoning helps the model make and update plans and cope with surprises, and the actions let it reach outside sources, such as a knowledge base, for facts it does not have.
For the Copilot, a ticket question looks like this:
Reason: "I need the status of ticket T-1042."
|
Act: call get_ticket("T-1042")
|
Observe: "open, assigned to IT support"
|
Reason: "I can answer now." -> Reply: "T-1042 is open and with IT support."
Compare it with step-by-step reasoning, which writes thoughts and nothing else. ReAct adds the act and observe steps, and the loop repeats until the model can answer. You have already used the act step: in helpdesk_agent.py the for step in range(1, 7) loop asks the model, runs any tool it asks for (read_policy or get_ticket), and asks again. This session gives a preview, so that the word is not new; session 20 builds the loop properly.
What is prompt chaining, and how does the helper split its job into two calls?
A support lead gets a long ticket and has two jobs: understand it, then reply to it. Given one prompt that does both, the reply can promise something the ticket never said, and nobody can tell where it came from. Split the job in two and the answer is easier to find: a first call summarises the ticket, and a second call answers from the summary.
That is prompt chaining: splitting a job into steps, each its own model call, where one step’s output becomes the next step’s input. It is an assembly line, where each station does one job and hands the part on. It is also a relay race, where each runner carries the baton a short distance.
Ticket T-1047 (from tickets.csv)
|
[ Call 1: Summarise ] model_prompts/summarise.md
| two sentences, no new facts
v summary
[ Call 2: Reply ] model_prompts/system.md
| working, tools, then JSON
v
answer, section, needs_human
Call 1 is plain: your code fetches the ticket, and the model only summarises it, with no tools. The prompt in model_prompts/summarise.md asks for two sentences saying who is asking, what they need and the ticket’s status, with no added facts. These lines of helpdesk_agent.py carry the summary out of call 1:
summary = None
if ticket_id:
ticket_text = get_ticket(ticket_id)
...
{"role": "user", "content": ticket_text},
...
summary = response.choices[0].message.content or ""
print("\nSummary of", ticket_id + ":", summary)
The print is on purpose: it shows you what passes between the calls. Call 2 puts the summary and the question into one message and then runs the tool loop from session 5:
user_message = ("Ticket summary: " + summary + "\n\n" if summary else "") + "Question: " + question
messages = [
{"role": "system", "content": (PROMPTS / "system.md").read_text()},
{"role": "user", "content": user_message},
]
The summary is optional. With no arguments the script uses the default question and ticket T-1047. If you type only a question, there is no ticket, so call 1 is skipped and call 2 gets the question alone.
Run it with python helpdesk_agent.py. The default question is “I have 7 unused leave days and ticket T-1047 is open. How many days can I keep?”. One real run printed this (your wording will differ, because replies vary):
Question: I have 7 unused leave days and ticket T-1047 is open. How many days can I keep?
Summary of T-1047: A user is asking how many of their 7 unused leave days they can carry over to next year. The ticket is currently open.
model asked: read_policy('leave')
(no usable tool call with tool_choice='auto'; asking again with less)
answer: The policy states that up to 5 unused annual‑leave days may be carried over into the next leave year. Since you have 7 unused days, you can keep 5 of those days; the remaining 2 will be forfeited.
section: Leave Policy 4.2
needs_human: False
Read it in order: the summary, then the tool line (model asked: read_policy('leave')), then a line in brackets, explained in the next section, then the answer, which starts with the model’s short working. A chain has trade-offs, shown below.
| One big prompt | A two-call chain |
|---|---|
| Summary and reply are produced together | The summary is a visible, printed step |
| A bad reply could come from anywhere | Read the summary first, then the reply |
| One instruction block to tune | Each call has its own prompt file |
| One call, tokens counted once | Two calls, so the second call’s input tokens count too |
The second call’s input is the summary, so a wrong summary becomes a wrong reply. The chain costs a second call and buys a place to look when something goes wrong.
Why can a chain lose information, and what does the line tool_choice do?
Failure first. In testing, the chain was run without the new line. Call 2 got the summary (“carrying over 7 unused leave days, ticket open”) and the question, and it answered straight away, with no tool call. The reply cited a made-up section, “Leave Policy 2.1”. It had never read the policy, because nothing in the summary said what the policy says.
The cause is the hand-over. Think of a message passed down a line of colleagues: by the last person, the detail has gone. Or a doctor who reads only the nurse’s note: the note says “carry-over question”, not what the rule is. A summary keeps what the person asked and leaves out most of the rest, including the policy, which was not in the ticket. Call 2 then fills the gap with something plausible.
The fix is one line in the tool loop of helpdesk_agent.py:
for step in range(1, 7):
message = ask_model(messages, "required" if step == 1 else "auto")
On the first turn (step == 1) tool_choice is "required": OpenAI’s function calling guide says this makes the model call one or more functions. After that it is "auto", which lets the model call zero, one or several, so it can stop calling tools and write its reply. With the line in place the model read the policy on the first turn in each run we made.
ask_model() makes the call and passes that value on as tool_choice. Some providers (Groq, for one) refuse the whole reply when the model calls a tool that is not in TOOLS, or calls none when one is required. So ask_model() steps down and asks again:
choices = ["required", "auto", None] if tool_choice == "required" else ["auto", None]
for choice in choices:
"required" becomes "auto", and "auto" becomes no tools at all, so the model can still answer in text. That is the bracketed line in the run above: on the second turn the model had read the policy, the "auto" call was refused, and the call with no tools gave the answer. If every try is refused, the helper does not guess; it sets needs_human to true and hands the question to a person. In the same way, the code runs only a tool listed in TOOL_FUNCTIONS, and sends “no such tool” back for any other name.
The prompt says the same in words: model_prompts/system.md states that the ticket summary is context for that ticket, never proof of the person’s own data, and never holds the policy, and that the model must call read_policy before it answers a question about a rule or a figure. It adds that a carry-forward limit is not anyone’s leave balance: asked for a personal balance that no tool returns, the model says it is unavailable, points to HR and sets needs_human to true. A chain passes mistakes along as well as facts, which is why you print between the calls.
What are three situations where a prompt lets you down, and what does each mean for the design?
- A chain answers a leave question from the summary alone and cites a section it never read. The summary holds the question, not the policy; the model must be made to call
read_policy. - The Copilot answers a carry-over sum with a confident wrong total. The answer came with no working to check; the “Think step by step” part puts the working at the start of
answer. - A ticket is summarised as closed when it is open, and the reply repeats it. Call 2 trusted a wrong summary; printing the summary shows the slip.
| What you learned | The design decision | Where it shows |
|---|---|---|
| A bare answer hides slips | Ask for the working on multi-step questions | model_prompts/system.md |
| One line of reasoning commits early | Try options where a wrong answer is costly | Your basic assignment |
| ReAct adds actions and observations | Preview now; build it as an agent later | Session 20 |
| A summary loses what it did not carry | Force a tool call on the first turn | tool_choice in helpdesk_agent.py |
| One big prompt is hard to debug | Split a long job into chained calls and print between them | summarise.md and system.md |
Project step: the help desk helper as a chain. Turn your session 5 helpdesk_agent.py into a chain of two calls. The build prompt to paste into Claude Code or Codex is prompts/1-write-helpdesk-agent.md. Keep both prompts in model_prompts/, print the summary between the calls, and use tool_choice as above. Run it on the default question, then run python helpdesk_agent.py "What is the status of this ticket?" T-9999: for a ticket that does not exist, call 1 should say it was not found and call 2 should set needs_human to true. Then change one thing and see what happens: delete the “Think step by step” part and run again, or make the summary wrong on purpose by adding “Say the ticket is closed.” to summarise.md. Stop when you have seen both changes.
Practice: explain and sort. First, answer the Model answer question below aloud in four beats before reading it. Second, for each of these four situations, decide whether the fix is the working, several options, a chain or a forced tool call, and write your reason in one sentence:
- A reply gives the wrong total for leave left.
- A reply to a ticket promises something the ticket does not say.
- A choice between three routes keeps picking the first one.
- A reply names a policy section the model never read.
Flashcards: step-by-step, Tree of Thoughts, ReAct and chaining questions with answers
Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.
1. What is step-by-step reasoning?
It means getting the model to write its intermediate steps before the final answer. Like showing your working in a maths exam, the steps expose a slip that a bare answer hides.2. When should you ask for the working, and when not?
Ask on questions with several steps, such as a carry-over sum, where a slip would otherwise be hidden. Skip it on plain lookups, and check the guidance for reasoning models, which says asking them to think step by step is unnecessary.3. Where does the working go when the helper's reply is JSON?
At the start of the `answer` field, as `model_prompts/system.md` asks, with the final answer after it. The working stays available for you to check, and the reply is still valid JSON.4. What is Tree of Thoughts, in plain words?
The model proposes several lines of reasoning, checks each one, keeps the promising ones and carries on from there, instead of following a single line. It is like a chess player who considers three moves before playing one.5. How does Graph of Thoughts differ from Tree of Thoughts?
It treats the model's thoughts as the points of a graph, with links showing which depends on which, so thoughts from different branches can be combined or improved by feedback. A tree keeps the best branch; a graph can merge parts of several.6. What does a tree of options cost, and when is it worth it?
It uses more tokens and sometimes more calls than a plain answer. Use it where a wrong answer is costly and there are several routes, and count the extra tokens before you decide.7. What is ReAct?
A pattern in which a model writes a reasoning step, takes an action such as calling a tool, reads what came back and reasons again. Step by step writes thoughts and nothing else; ReAct adds the actions and observations. Session 20 builds it.8. What is prompt chaining?
It splits a job into steps, each its own model call, where one call's output becomes the next call's input. Like an assembly line, each station does one job and hands the part on.9. Why does the helper print the summary between the two calls?
A chain passes mistakes along: a wrong summary becomes a wrong reply. Printing what passes between the calls lets you see which of the two calls went wrong.10. What does `ask_model(messages, "required" if step == 1 else "auto")` do, and why is it there?
On the first turn of call 2 it makes the model call a tool, and after that it lets the model decide. If the provider refuses that call, `ask_model()` asks again with less: `"auto"`, then no tools. Without it, the chain answered from the summary alone and cited a made-up section, because the summary holds the question and not the policy.11. What is the cost of a chain compared with one prompt?
A chain makes a second call, so the second call's input tokens count too. In return you get a visible step to inspect when something goes wrong.Model answer: how would you change the prompts so the Copilot shows its working and handles a long job?
The core question this session answers: “A teammate asks: the Copilot gave a confident wrong total on a carry-over sum, and one long prompt both summarises a ticket and writes the reply. How would you change the prompts?”
Four beats, in this order: ask for the working · try several routes when it matters · chain the steps · read each step. Skip any of these four and the answer is incomplete.
- Ask for the working: work through the question, find the section, apply it to the numbers, then answer.
- Try several routes when it matters: a tree of options, as an idea, costs more calls and helps on harder problems.
- Chain the steps: summarise first, then reply from the summary, each call with its own prompt file.
- Read each step: the summary, the tool line and the working are where a slip shows.
The full answer, said aloud:
(Ask for the working) “For the sum, I would ask the model to work through the question in order: what it asks, which policy section covers it, and what the section says about the employee’s numbers, then answer. If the total is wrong, the line where it went wrong is on the page.
(Try several routes when it matters) Where a question has several routes and a wrong answer is costly, I would ask for three options and what goes wrong with each, then keep the best. That is the Tree of Thoughts idea, and I would count the extra tokens before using it.
(Chain the steps) For the ticket, I would split the prompt into two calls: one summarises, and a second answers from the summary and must read the policy, so it cannot answer from the summary alone.
(Read each step) Then I print and read the summary, the tool call and the working first, because that is where a slip becomes visible.“
Missing a beat is a failure state. If you stop before the last beat, the follow-up is how you find which step went wrong. Practise until the four come out without prompting.
Why this matters: where do you use step-by-step prompts and chains at work?
In a review of a wrong answer: the first question is where the answer went wrong. A reply with its working, or a chain with a printed summary, lets you point at the step instead of arguing about the whole.
In a design conversation about reasoning: names such as chain of thought, Tree of Thoughts and ReAct appear in articles and vendor talks. Knowing what each adds, and what it costs, lets you decide when a plain prompt is enough.
When a multi-step assistant gives a confident but unsupported answer: a chain can drop the very fact the last step needs. Checking what passes between calls, and forcing a lookup where the answer must come from a source, is the same habit in any team.
In the rest of this course: session 8 tests the helper against a test set and keeps its prompts in a library; session 10 chains the sorting of a question before its answer; session 20 builds the ReAct loop as an agent; and the evaluation sessions score reasoning with a model that grades answers.
After this unit: what is the assignment and what comes next?
Assignment. Solve one puzzle step by step and as a tree of options, and compare the two replies. The brief for the project work, the help desk helper as a chain of two calls, is the project file. Keep your chain: the next session runs a test set against the helper.
What’s next. Session 8 is prompt engineering 3: breaking a prompt with one bad input, testing it against a test set with a pass count, and keeping named, versioned prompts in a prompt library.
Want help building this with someone?
Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.
Ask about mentor-led trainingTechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.