Courses / Agentic AI from Scratch (Python)

Session 10 of 40

Prompt chaining in LangChain: sort the question first, then answer it

Notes

How to Route Questions With Prompt Chaining in LangChain: Sort the Question, Then Answer From One Policy

Agentic AI Foundations · Session 10 — Prompt chaining in LangChain: sort the question first, then answer it

Before you start: what do you need to build a two-step chain?

You need the code folder for this session, chapters/10-sort-then-answer/, open in your terminal. It holds helpdesk_router.py, run_test_set.py, two prompts in model_prompts/ (sort.md and answer.md), data/policies/ with the three policies, data/test_set.json with eight test cases, and a build prompt in prompts/. Have your environment switched on, your three settings set as in session 1, and the libraries from session 9 installed (pip install langchain-core langchain-openai pydantic). Nothing new to install. Have paper or a notes app ready for the Flashcards and the Model answer.

Summary: how do you sort a question before answering it?

Session 9 put all three policies into one prompt, so a question about no policy still reached the answerer and every answer was read against all the text. This session splits the helper into two chains. The first chain only sorts the question into one label from a fixed list: leave, it-security, expenses or other. Plain Python then uses the label to open one policy file, and the second chain answers from that policy alone. A question labelled other goes to a person without a second model call, because your code decides that, not the model. The test runner now prints the label beside each case, so a failure tells you which prompt file to fix.

Key concepts: prompt chaining, routing, sorter, label, Literal and needs_human defined

Term Definition
Prompt chaining Splitting a job into steps, where one model call works on the output of the previous step, so each step is smaller and can be checked.
Routing A first step that decides where an input should go, so each kind of input gets the handling that suits it.
Sorter The first chain of the helper: it reads the question and returns one label and nothing else.
Label One value from a fixed list (leave, it-security, expenses or other) that says which policy a question is about.
Literal A Python type that lists the exact values a field may hold; here it lists the four labels.
other The label for a question that fits none of the three policies; it goes to a person.
needs_human The reply field that says a person must take over; ask_helper sets it to true when the label is other.
sort.md and answer.md The two prompt files: the sorting prompt, and the answering prompt with a {policy} blank.

Explanations: how do the sorter, the label and the answerer work together?

Where are we? What does this session add to the HelpDesk Copilot?

A reply follows five steps: pick the policy section, build the request, read it as tokens, write the reply token by token, and give the answer. Today changes steps 1 and 2. A first call sorts the question and your code picks the policy (step 1); a second call builds and sends the request with that one policy (step 2). The code is chapters/10-sort-then-answer/, and helpdesk_router.py keeps the function ask_helper(question) from session 9.

What is prompt chaining, and why sort the question first?

A hospital triage desk is the picture. A nurse asks what is wrong and decides which department you need before you see anyone. A post-room sorter is the same: each letter goes to the right pigeonhole before delivery. Prompt chaining is splitting a job so that one call works on the output of the previous step. LangChain’s guide on workflows describes it as each call processing the output of the previous call, often used for well-defined tasks that can be broken into smaller, verifiable steps. Routing is the first step of such a chain: it works out the kind of input and sends it to the task that suits it.

Why do it when session 9 worked? The answerer now reads one policy, not all of them, so it has less text to be confused by, and a question about no policy never reaches it. The cost is a second model call for every question that is not labelled other.

What does the question sorter return, and why a fixed list?

Think of a multiple-choice sheet: you tick one box and do not write an essay. The sorter is a first chain with one job, returning one label. Its reply has one field, and Literal[...] lists the four values it may hold, so your code can rely on it:

class Sorted(BaseModel):
    label: Literal["leave", "it-security", "expenses", "other"] = Field(
        description="Which policy the question is about, or other."
    )

A parcel label with four destinations works the same way: the machine reads the label, not the contents. A fixed list turns a fuzzy judgement into something code can branch on. The sorter’s prompt, model_prompts/sort.md, describes each label in one line, says that other covers questions about any policy that is not one of the three (such as “the travel policy” or “the sales policy”) and questions not about the company at all, and ends: “If you are not sure, give other.”

How do two prompts and two chains fit together?

Each step has its own prompt file and its own chain, built the way you built the chain in session 9: a template, the pipe, a typed reply.

Step Prompt file What the model sees
Sort model_prompts/sort.md The question only
Answer model_prompts/answer.md The question and one policy, in the {policy} blank
sort_chain = sort_prompt | model.with_structured_output(Sorted)
answer_chain = answer_prompt | model.with_structured_output(HelpDeskAnswer)

Each chain then gets the same safety net as in session 9: when the provider refuses a typed reply with a 400 error, with_fallbacks asks once more with the reply as a tool call. For the sorter it reads:

sort_chain = sort_chain.with_fallbacks(
    [sort_prompt | model.with_structured_output(Sorted, method="function_calling")],
    exceptions_to_handle=(BadRequestError,),
)

answer_chain gets the same lines with answer_prompt and HelpDeskAnswer. The chat model is built exactly as in session 9, including the check that stops with one line when there is no API key.

The answer prompt keeps two rules from session 9: answer only from the policy text without inventing a figure or a section number, and set needs_human to true, saying what is missing, when the section asked about is not in the policy text or the policy text does not answer the question.

How does plain Python join the chains, and what happens to an “other” question?

A receptionist with a clipboard reads the visitor’s reason, ticks a box and walks them to the right room. The receptionist does not answer the question. A railway signal is the same: the signal says where, and the railway moves the points. Here the model reads and labels, and plain Python takes the action. This is the whole of ask_helper’s logic:

label = sort_chain.invoke({"question": question}).label
if label == "other":
    return {"answer": "This is not a question I can answer ...",
            "section": "none", "needs_human": True, "label": label}
policy = (DATA / "policies" / f"{label}-policy.md").read_text()
reply = answer_chain.invoke({"question": question, "policy": policy}).model_dump()
 Question --> 1. Sort chain --> label
                                  |
     other --> person (needs_human = True, no second call)
     leave | it-security | expenses
                                  |
                  read that one policy file
                                  |
              2. Answer chain --> answer, section, needs_human

The model never chooses a file; your code does, from the label. So an other question stops after one model call with needs_human true. Try python helpdesk_router.py "What is the capital of France?": the label is other, needs_human is true, and the second call never happens. The function still returns a dictionary with the three fields plus the label, or None if a chain failed.

How do I run it, and how do I read a failure by its label?

Run the helper once:

python helpdesk_router.py

A real run printed this; your wording will differ:

Question: I have 7 unused leave days. How many days can I keep?
label: leave
answer: Up to 5 unused annual-leave days carry over into the next leave year.
section: Leave Policy 4.2
needs_human: False

Then run the test set, which now prints the label the sorter chose beside each case:

python run_test_set.py
PASS - carry-over days [leave]
PASS - who approves leave [leave]
PASS - lost laptop [it-security]
PASS - claiming an expense [expenses]
PASS - section that does not exist [leave]
PASS - policy that does not exist [other]
PASS - sick leave allowance [leave]
PASS - off-topic question [other]

Passed 8 of 8

The label in square brackets is how you read a failure. A wrong label means fix sort.md; a right label and a wrong answer means fix answer.md. Run it a few times, since replies vary. If something fails, check this table:

You see Do this
ModuleNotFoundError: langchain_openai Run the pip install line from session 9, with your environment switched on.
An error with 401 The service did not accept the key (a typing slip, or a key that is no longer valid); check your settings as in session 1. With no key at all, the program stops earlier with No API key.
A case fails with the wrong label Open model_prompts/sort.md and add a line for that kind of question.
FileNotFoundError for a policy The label must match a file name: leave, expenses or it-security.

What are three situations where sorting goes wrong, and what does each mean for the design?

  1. A case fails and its label in square brackets is wrong. The sorter prompt has no line for that kind of question, so add one to sort.md.
  2. A question fits two policies, such as whether a home-office chair can be claimed as an expense. The sorter must pick one label, so the answerer reads only one policy. Ask it and see which label comes back.
  3. You add a new label such as hr-payroll to Sorted. Its questions fail until the answer step has a policy file with a matching name, data/policies/hr-payroll-policy.md, and the sorting prompt has a line for it.
What you learned The design decision Where it shows
A wrong label is a sorting problem Read the label first; fix sort.md The [label] in the test output
A right label with a wrong answer is an answering problem Fix answer.md The same test output
A question that fits no policy has no business with the answerer Send other to a person with no second call if label == "other"
A new kind of question needs a policy and a sorting line Add the file, the label and the prompt line together Sorted, sort.md, data/policies/

Basic assignment: build the two-step helper, run the test set, and write down any question that was sorted wrongly.

Project step: build the router. Write helpdesk_router.py with your coding assistant using prompts/1-write-helpdesk-router.md. run_test_set.py changes only by one import and one printed label, so keep your session 9 prompt for it and ask for those two edits. Run both. Remove the line about “the travel policy” from sort.md and run the test set to see which case moves. Then ask a question that fits two policies and write down which label the sorter picks. Stop when you have a pass count and one wrongly sorted question written down.

Practice: explain and sort. First, answer the Model answer question below aloud in four beats before reading it. Second, for each of these four problems, say whether you would edit sort.md, answer.md or a policy file, and why: (1) a laptop question labelled leave; (2) a right label with an answer that invents a section; (3) a new payroll policy that needs its own label; (4) a question about the sales policy that the answerer tried to answer.

Flashcards: prompt chaining, routing, sorter and label questions with answers

Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.

1. What is prompt chaining? It splits a job into steps, where one model call works on the output of the previous step. Each step is smaller and can be checked on its own, like a triage desk that decides the department before the specialist sees you.
2. What is routing, and which step is the routing step in the helper? Routing is a first step that decides where an input should go. In the helper it is the sort chain, which labels the question before anything is answered.
3. What does the sorter return, and why is the list fixed? One label: `leave`, `it-security`, `expenses` or `other`. `Literal` lists the four values, so the sorter cannot reply with a label you did not plan for, and your code can branch on it.
4. Who picks which policy file the answerer reads? Your code does. The model returns only the label, and plain Python reads `data/policies/
5. What happens to a question labelled `other`? `ask_helper` returns `needs_human` true with a fixed message and the second model call never happens, so a question about no policy never reaches the answerer.
6. Why sort first, when session 9 put every policy in one prompt? The answerer now reads one policy, not all of them, so it has less text to be confused by. The cost is a second model call for each question that is not labelled `other`.
7. What does the sorter's prompt do with a question about "the travel policy"? `sort.md` says `other` covers a question about any policy that is not one of the three, with the travel policy as an example. A question about it is labelled `other` and goes to a person.
8. How do you read a failing case with the label printed? A wrong label means fix `sort.md`. A right label with a wrong answer means fix `answer.md`. The label in square brackets tells you which prompt file to open.
9. What changes in `run_test_set.py` from session 9? One import, from `helpdesk_router` instead of `helpdesk_chain`, and one printed label beside each result. The three checks and the pass count are the same.
10. What must you add to give the helper a fourth policy? A policy file whose name matches the new label, the label itself in `Sorted`, and a line for it in `sort.md`. Without a matching file, questions with that label fail.

Model answer: why does the helper make two model calls, and what does your code do between them?

The core question this session answers: “Why does the helper make two model calls, and what does your code do between them?”

Four beats, in this order: sort · pick the policy in code · answer from one policy · send the rest to a person. Skip any of these four and the answer is incomplete.

  1. Sort: the first chain reads the question and returns one label from a fixed list: leave, it-security, expenses or other.
  2. Pick the policy in code: plain Python reads data/policies/<label>-policy.md; the model never chooses a file.
  3. Answer from one policy: the second chain answers from that policy only, as a typed reply.
  4. Send the rest to a person: a label of other returns needs_human true with no second model call.

The full answer, said aloud:

(Sort) “The first call has one job: read the question and return one label from a fixed list, leave, it-security, expenses or other.

(Pick the policy in code) My code takes the label and opens that one policy file. The model does not choose a file; it only labels, and plain Python acts on the label.

(Answer from one policy) The second call answers from that single policy as a typed reply, so it has less text to be confused by than when every policy was in the prompt.

(Send the rest to a person) If the label is other, the code returns a reply with needs_human true and never makes the second call, so a question about no policy is handed over, not guessed.“

Missing a beat is a failure state. The follow-up question lands where you went quiet: skip the last beat and the next question is what happens to a question that fits no policy. Practise until the four come out without prompting.

Why this matters: where do you use routing and prompt chains at work?

In a design review: someone asks why the assistant does not use one big prompt. Two small jobs with a label between them are easier to test and to fix than one large one, and you can say which prompt file to open when a case fails.

In a risk review: a reviewer asks who decides which document the model reads. Your answer is that the code does, from a fixed list of labels, and anything outside the list goes to a person.

When you plan the next feature: a product owner wants a fourth kind of question. You know the three things to add: a policy file, a label and a line in the sorting prompt.

In the rest of this course: session 11 gives this two-step helper a memory, and the agent sessions later reuse the same shape, a label that decides what happens next.

After this unit: what is the assignment and what comes next?

Assignment. Build the two-step helper, run the test set, and write down any question that was sorted wrongly. The brief for the project work is the project file. Keep helpdesk_router.py: the next session turns it into a chat.

What’s next. Session 11 adds memory: a chat loop keeps the conversation so a follow-up question is understood.

Want help building this with someone?

Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.

Ask about mentor-led training

TechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.