Courses / Agentic AI from Scratch (Python)

Session 5 of 40

Structured Outputs and function calling: replies a program can trust

Notes

How to Get Reliable JSON From an LLM: a Fixed Reply Format, json.loads and Function Calling in Python

Agentic AI Foundations · Session 5 — Structured Outputs and function calling: replies a program can trust

Before you start: what do you need to use a fixed reply format and function calling?

You need the setup from the first sessions: Python, the openai package, a model endpoint that accepts the OpenAI request format, and your three settings (LLM_BASE_URL, LLM_API_KEY and LLM_MODEL). You can type them between the quotes at the top of helpdesk_agent.py or set them in your terminal; if a setting is in the terminal, the terminal wins. Recap of what this session assumes: a Chat Completions request is a list of messages, each with a role (system, user or assistant), sent to the model, and the reply comes back as the next assistant message. The code folder holds helpdesk_agent.py, a data/ folder with three policies and data/tickets.csv, and a prompts/ folder. Run every command from the folder that holds helpdesk_agent.py, so the data folder sits next to it. The model must support function calling; check your provider’s documentation. Have paper or a notes app ready for the Flashcards and the Model answer.

Summary: how do you make an LLM reply a program can read, and let the model use a function?

Up to now the Copilot’s replies were written for a person to read, and a person chose which policy section the model saw. A program that reads a reply breaks as soon as the reply is a sentence, is cut short or renames a key. So the helper’s prompt asks for JSON only, with exactly three keys, answer, section and needs_human, and the code reads the reply with json.loads, says plainly when it is not valid JSON, and then reads the three keys by name. Function calling goes one step further: the model replies with a request to call a function you described, your code runs it, and the result goes back so the model can write the final reply. In helpdesk_agent.py the model uses it to read the policy file and the ticket it needs, while a fixed format fixes the shape of a reply and not whether what it says is true.

Key concepts: reply format, JSON, json.loads, function calling and tool defined

Term Definition
Fixed reply format A reply that is meant to have the same named fields every time, so a program can read it instead of a sentence. In the helper: answer, section and needs_human.
JSON A text format for named values, such as {"answer": "Up to 5 days", "needs_human": false}, that programs can read.
json.loads The Python call that turns valid JSON text into the matching Python value (an object becomes a dictionary, an array a list, null becomes None), and raises JSONDecodeError only when the text is not valid JSON. It does not check which keys are present.
needs_human The helper’s true or false field: true when a tool finds nothing or the policy does not cover the question, so a person should take over.
Function calling The model asks your code to run a function: it replies with the function’s name and arguments, your code runs it, and you send back the result.
Tool The general name for something a model can ask for; a function is a kind of tool. The helper has two: read_policy and get_ticket.
Tool description The name, the purpose and the needed arguments of a tool, written in a list called TOOLS; it is all the model knows about the tool.
Tool call The model’s request to run a tool, with an id, a name and JSON-encoded arguments.
The loop The repeated cycle of asking the model, running any tool it asks for, adding the result and asking again, until the model answers.

Explanations: how do a fixed reply format, json.loads and function calling work together?

Where are we? What does this session add to the HelpDesk Copilot?

In session 3 a person picked the policy section, and in session 4 a person still pasted it. This session moves step 1 of the reply, “Pick the policy section”, to the model: it asks your code to read the policy file. The new code is one file, helpdesk_agent.py, with the data it reads in data/:

Box What it does for the employee
read_policy(name) Returns the text of one policy file, such as the leave policy
get_ticket(ticket_id) Returns one row of data/tickets.csv as JSON text
TOOLS Tells the model the two tools exist and what each needs
The loop Runs the tools the model asks for and sends back each result
The JSON check Reads the final reply with json.loads and prints the three fields

Read it as ask, run, answer: the model asks, your small functions do the reading, and the reply is checked before it is printed.

What does the helper do when you run it?

Run the helper before reading how it works:

python helpdesk_agent.py

The default question is “I have 7 unused leave days and ticket T-1047 is open. How many days can I keep?” A real run on 2026-11-11 printed this:

Question: I have 7 unused leave days and ticket T-1047 is open. How many days can I keep?
model asked: read_policy('leave')
model asked: get_ticket('T-1047')
answer: You can carry over up to 5 unused annual-leave days into the next leave year.
section: Leave Policy 4.2
needs_human: False

Your run can differ from this one: the answer’s wording may change from run to run, and the model may ask for the two tools in either order. What you should expect is the shape: some model asked: lines, then three fields. Nobody pasted a policy into the program. The model asked for the leave policy and for ticket T-1047 by name, and the program fetched them.

What are the policy files the program reads?

Think of a filing cabinet: the model is at a desk with no key, and a clerk fetches any folder it names. Or think of a library desk: you ask for a book by title and the librarian brings it. In session 5 your program is the clerk. Before this session you pasted policy text into the script yourself; from now on the program reads the files. They are ordinary text files, so open one:

cat data/policies/leave-policy.md       # Mac
type data\policies\leave-policy.md      # Windows

You will see headings such as ## 4. Annual leave and numbered lines such as 4.2 Up to 5 unused annual-leave days carry over into the next leave year. Open data/tickets.csv the same way: one ticket per line, with the columns id, title, description, category, status and assignee, and T-1047 is the leave question. The model cannot open these files. It can only ask, and that is what the next section describes.

What is function calling, and how does a model use a tool without running it?

Picture a manager and an assistant. The manager says, “Look up ticket T-1047 and tell me where it stands.” The assistant does not guess: they go to the system, read the ticket and report back. Or picture a restaurant order slip: the waiter writes it and the kitchen cooks. In function calling, the model writes the slip and your code is the kitchen.

Failure first. An employee asks, “Raise a ticket for my broken laptop.” The model replies, “Done, I have raised a ticket.” No ticket exists. A reply is just text; nothing ran. The model had no way to reach your ticket system, and nothing told it that it could ask.

OpenAI describes tool calling as a back-and-forth between your application and the model with five steps: your first request lists the tools the model may use; the model answers with a tool call; your code runs the function with the arguments from that call; a second request carries the tool’s output; and the model gives its final reply, or asks for more calls. A function is one kind of tool. Amazon Bedrock’s documentation describes the same split for its own models: the model decides when a tool is needed, but it does not directly call the tool; your application code runs it and returns the result for the model to use in its final response.

 1. Request with tools   ->  model
 2. Model replies: "call read_policy with name leave"
 3. Your code runs read_policy('leave')
 4. Next request: messages + the tool output (role "tool")
 5. Model asks again, or writes the final reply

Each model asked: line in the run above is one trip round this loop.

How do you describe a tool, and how does the loop run?

Think of a job card pinned beside a tool in a workshop: its name, what it is for and what you must bring. Or a menu entry: a dish name, a short description and the choices you must make. The model sees only the card. Here is the description of get_ticket in helpdesk_agent.py:

{
    "type": "function",
    "function": {
        "name": "get_ticket",
        "description": "Look up one support ticket by its id, for example T-1042.",
        "parameters": {
            "type": "object",
            "properties": {"ticket_id": {"type": "string"}},
            "required": ["ticket_id"],
        },
    },
},

The description for read_policy has the same shape. Its name argument must be one of leave, expenses or it-security, listed as an enum. A vague description gives vague tool calls, so write it as if for a new colleague.

The functions themselves are ordinary code. read_policy builds the path data/policies/<name>-policy.md and returns the text, or no such policy when the file is missing. get_ticket reads data/tickets.csv and returns the matching row as JSON text, or ticket not found. Nothing about them is special, which is the point: OpenAI describes a function as a kind of tool defined by a JSON schema, so you can describe a function of your own in the same shape, and Module 5 builds on it.

The loop is the heart of the program:

for step in range(1, 7):
    response = client.chat.completions.create(
        model=os.environ.get("LLM_MODEL", LLM_MODEL),
        messages=messages,
        tools=TOOLS,
    )
    message = response.choices[0].message
    if not message.tool_calls:
        break
    messages.append(message)
    for call in message.tool_calls:
        arguments = json.loads(call.function.arguments)
        value = list(arguments.values())[0]
        print(f"model asked: {call.function.name}({value!r})")
        if call.function.name == "read_policy":
            result = read_policy(value)
        else:
            result = get_ticket(value)
        messages.append({"role": "tool", "tool_call_id": call.id, "content": result})

Read it in order. The loop runs at most six times, so a confused model cannot loop forever. If the reply has no tool_calls, the model has answered and the loop stops. Otherwise the model’s message is added to the list, and each call is handled: its arguments arrive as JSON text, so json.loads turns them into a dictionary; your code prints the model asked: line and runs the right function; and the result goes back as a message with the role tool and the call’s id, so the model knows which request it answers. OpenAI’s guide also says model responses can include zero, one or several calls, which is why the code goes through every call in message.tool_calls.

Why does a program need a fixed reply format, and how does json.loads check it?

Picture a spreadsheet with a date column. A person writes “next Tuesday-ish” on a note and reads it fine; the spreadsheet rejects it. Or picture a mail-sorting machine: it reads the postcode box, and an address written round the edge of the envelope is lost. A person reads meaning from any layout. A program reads a name or position it was told to expect, and when the layout moves, the program does not notice; it breaks.

Failure first, then cause. Your script expects an object like {"answer": "...", "section": "4.2"} and meets three different replies:

Your script expects:   {"answer": "...", "section": "4.2"}

Reply A (a sentence):  Up to 5 days carry over, see Leave Policy 4.2.
Reply B (cut short):   {"answer": "Up to 5 days", "section": "4.2"
Reply C (renamed):     {"answer": "Up to 5 days", "sections": ["4.2"]}

Reply A has no named field, so the script must guess where the section is. Reply B is missing its closing brace, so it is not valid JSON. Reply C is valid JSON, but the key is sections, so code that reads section fails. Asking for JSON politely does not prevent these three. So the helper does two things.

First, it asks precisely. This is the reply format in the helper’s prompt (the system_message), in short: reply with JSON only and no other text, with exactly three keys. answer is a short string. section is the policy section used, named by policy and number, such as “Expenses Policy 3.2”, or none. needs_human is true or false, and it is set to true when a tool finds nothing, when the policy does not cover the question, or when the question is not about help desk topics.

Second, it checks. A prompt is a request, not a rule the reply is held to, so the code reads the reply itself:

reply = (message.content or "").strip()
text = reply.removeprefix("```json").removeprefix("```").removesuffix("```").strip()
try:
    data = json.loads(text)
except json.JSONDecodeError:
    print("The reply is not valid JSON, so a program cannot use it:", reply)
else:
    print("answer:", data["answer"])
    print("section:", data["section"])
    print("needs_human:", data["needs_human"])

A model may wrap JSON in a code fence (three backticks) even when told to reply with JSON only, so the code removes a fence first, in case there is one. Then json.loads turns the text into a Python value, or raises JSONDecodeError if the text is not valid JSON, and the program says plainly that the reply is not valid JSON and shows it. json.loads does not check which keys exist, and neither does this script: the else branch simply reads data["answer"], data["section"] and data["needs_human"] by name. A reply that is valid JSON but lacks one of those keys (like Reply C above) makes that line raise a KeyError, and a valid JSON value that is not an object (a list, say) raises a TypeError; the script stops with an error instead of a friendly message. A stricter version would check the keys before reading them. Some providers also offer a feature that enforces a schema: according to OpenAI’s guide, its Structured Outputs feature constrains the model’s completed, non-refusal output to a JSON Schema you supply, and your application still has to handle refusals and check business rules the schema does not express. Whether a given compatible server enforces schemas is for its own documentation to say. This session uses the plain way, which needs no special server feature: ask in words, then check. How closely a given model follows the instruction is something you test, not assume.

A valid shape is still not a true fact. A perfectly shaped reply can hold an invented section:

{"answer": "Up to 5 unused days carry over.", "section": "Leave Policy 4.9", "needs_human": false}

Three keys, the right types, and a section that does not exist: the sample leave policy has no section 4.9. The policy files stay the source of truth, and a test set (session 8) is how you catch this.

What happens when the data cannot answer the question?

Imagine asking a clerk for a folder that is not in the cabinet. A good clerk says “it is not here”, not “here is a folder I made up”. Run a question the data cannot answer by typing it after the script name:

python helpdesk_agent.py "What is the status of ticket T-9999?"

Ticket T-9999 does not exist, so get_ticket returns ticket not found. A real run on 2026-11-11 printed:

Question: What is the status of ticket T-9999?
model asked: get_ticket('T-9999')
answer: I can't find a ticket with that ID. Please check the ID and try again.
section: none
needs_human: True

The model did not invent a status. The prompt told it to set needs_human to true when a tool finds nothing, and it did, with section: none. A program can now act on that flag, for example by sending the question to a person.

Basic assignment: run both commands above. For each, list the model asked: lines, then say in one sentence why the second run ended with needs_human: True. Then change one thing and see: ask “How many sick leave days do I get?”, or delete the words “Reply with JSON only and no other text” from system_message and run again. If the script prints “The reply is not valid JSON”, you have seen why a program needs a fixed format.

Project step: the Copilot’s fixed answer format. Paste the prompt in prompts/1-write-helpdesk-agent.md into Claude Code or Codex, with the data/ folder next to the new file, and run the result. Stop when python helpdesk_agent.py prints the tool steps and the three fields for the T-1047 question, and the T-9999 question prints needs_human: True.

What do these ideas mean for the design of the Copilot?

Before the design table, three situations you will meet:

  1. A script that files the cited section crashes on one reply in twenty. That reply was a sentence, not an object with a section key, so there was nothing to read.
  2. The assistant says “Done, ticket created” and no ticket exists. A reply is text; creating a ticket needs a function your code runs after the model asks.
  3. A reply has all three keys and cites a section that does not exist. A valid shape is not a true fact.

What these mean for the Copilot’s design:

What you learned The design decision Where it shows
Programs read named fields Ask for JSON with exactly three keys system_message
A reply may not be valid JSON Check with json.loads and say so plainly The try and except block
A valid shape can hold a wrong fact Keep the policy files as the source data/policies/
A reply cannot change anything Offer a function and run it in your code read_policy, get_ticket, TOOLS
The data may not hold the answer Let the tool say “not found” and set needs_human The T-9999 run

Practice: explain and sort. First, answer the Model answer question below aloud in four beats before reading it. Second, for each of these four situations, decide whether the fix is exact keys in the prompt, a json.loads check, a source check or a function call, and write your reason in one sentence:

  1. A reply renames a key and the script cannot find it.
  2. The reply is cut short and is not valid JSON.
  3. A valid object cites a section that does not exist.
  4. The model must report the real status of ticket T-1047.

Flashcards: reply format, json.loads and function calling questions with answers

Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.

1. Why does free text break a program? A person reads meaning from any layout, but a program reads a name or position it was told to expect. A sentence has no named field, a cut-off reply is invalid JSON, and a renamed key is not found, so the program fails.
2. What three keys does the helper's reply have? `answer`, a short string; `section`, the policy section used, such as "Leave Policy 4.2", or `none`; and `needs_human`, true or false.
3. What does json.loads do, and what if the reply is not valid JSON? It turns valid JSON text into the matching Python value (an object becomes a dictionary). If the text is not valid JSON it raises `JSONDecodeError`, and the helper prints that the reply is not valid JSON and shows the reply instead of guessing. It does not check which keys are present: the helper then reads the three keys by name, so a valid reply with a missing key stops the script with a `KeyError`.
4. Why does the code remove three backticks before json.loads? A model may wrap its JSON in a code fence. The fence is not JSON, so `json.loads` would fail on a reply that is otherwise fine; the code removes one in case it is there.
5. Does a valid shape mean a true answer? No. A reply can have the right keys and types and still cite a section that does not exist. Keep the policy files as the source and check against them.
6. What is function calling? The model replies with the name and arguments of a function you described, your code runs it, and you send the result back so the model can write the final reply. The model decides when a function is needed; it does not run anything itself.
7. What are the steps of the function-calling flow? Send a request with the tools described, receive a tool call, run the function in your code, send the next request with the tool output, and receive the final reply or more tool calls.
8. What does the tool description tell the model? A name, what the tool does and what arguments it needs. It is all the model knows about the tool, so a vague description gives vague calls.
9. Why does the loop run at most six times, and why handle several calls? The limit stops a confused model from looping forever. A model response can hold zero, one or several tool calls, so the code goes through every call and sends back a result for each call's id.
10. What does the helper do for ticket T-9999? `get_ticket` returns `ticket not found`, and the model sets `section` to `none` and `needs_human` to true, so a person can take over instead of the helper guessing.

Model answer: how do we make the Copilot’s replies safe for a program to read, and how does the model get our code to look something up?

The core question this session answers: “An interviewer asks: how would you make a model’s replies safe for a program to read, and how does the model get your code to look something up?”

Four beats an interviewer listens for, in this order: ask for exact keys · check with json.loads · a valid shape is not a true fact · function calling. Skip any of these four and the answer is incomplete.

  1. Ask for exact keys: the prompt says JSON only, with answer, section and needs_human.
  2. Check with json.loads: if the reply is not valid JSON, report it plainly.
  3. A valid shape is not a true fact: keep the policy files as the source.
  4. Function calling: the model asks for a function and its arguments, your code runs it, and the result goes back for the final reply.

The full answer, said aloud:

(Ask for exact keys) “A program needs named fields, not a sentence. The prompt asks for JSON only, with exactly three keys, so the reply has a place for each thing the program needs.

(Check with json.loads) A prompt is a request, so I read the reply with json.loads, and if it is not valid JSON I say so plainly instead of guessing.

(A valid shape is not a true fact) A reply can have the right shape and an invented section number, so the policy files stay the source.

(Function calling) To look something up, I describe a function. The model replies with its name and arguments, my code runs it, and I send the result back so the model can write the final reply.“

Missing a beat is a failure state. If you stop before function calling, the interviewer’s follow-up is how the model actually gets a ticket’s status. Practise until the four come out without prompting.

Why this matters: where do you use a fixed reply format and function calling at work?

In an integration meeting: whenever a model’s reply feeds another system, such as a ticket queue or a form, someone asks how you will stop a stray sentence from breaking it. A fixed format and a clear failure path are the answer.

In a design review of an assistant that acts: the split between the model asking and your code running is what lets you add checks, logging and approval before anything changes. That split is the base of the agents later in this course.

In the rest of this course: session 6 moves the helper’s prompt into a file and teaches its parts; module 3 uses typed replies in LangChain; module 5 turns function calling into agent tools; and module 6 adds a person’s approval before a risky function runs.

After this unit: what is the assignment and what comes next?

Assignment. Run the helper on the T-1047 question and the T-9999 question, then change one thing and see what happens. The brief for the project work, the Copilot’s fixed answer format, is the project file. Keep your helper: the next session moves its prompt out of the code.

What’s next. Session 6 starts prompt engineering: the parts of a prompt, zero-shot and few-shot examples, and a persona for the system message, with the prompt kept in a file.

Want help building this with someone?

Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.

Ask about mentor-led training

TechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.