Chapter 2 — LLM APIs and Structured Output

Course: GenAI and Agents Fast Track Prerequisites: basic Python (functions, dictionaries, lists, exceptions); what a token, a context window, temperature and a prompt are.

What the reader can do after this unit:

• Send a list of messages to a hosted or local model from Python and read the text, the finish reason and the token counts that come back
• Turn a model’s reply into a validated Python object with a schema, and retry or fail cleanly when the reply does not fit
• Let a model request a function call, run that function in your own code, return the result, and work out what the call cost from its token counts

Core question this unit answers: How do you get an answer from a hosted or local model that a Python program can read and act on without breaking?

Connections:

• Builds on: tokens, the context window, temperature and prompting, as a way to talk to a model by hand
• Leads into: giving the model passages from your own documents so it can answer from them and name the pages (Chapter 3)

Student Notes — Chapter 2: LLM APIs and Structured Output

GenAI and Agents Fast Track · Chapter 2

Before you start

You need Python 3 and a text editor. To run the code in these notes against a real model you also need an endpoint that speaks the OpenAI-style chat format: a hosted provider (which needs an account and a key, and may charge by the token) or a model server running on your own computer. Tasks 1 and 2 of the project need no model at all, so you can do most of the chapter with neither. Install the two packages used here with pip install openai pydantic.

Summary

This chapter shows how a Python program talks to a language model. A call is one request holding a list of messages and a few settings, and one response holding the text, a reason it stopped and the token counts. Hosted and local models can share the same request shape, so the same code works against both once three values change. Because a program cannot use a chatty paragraph, you ask for JSON, check it against a schema and retry when it does not fit. When a model needs facts or actions that only your code has, function calling lets the model ask and your code run the function, and you finish by working out what a call costs in input and output tokens.

Key concepts

Term	Definition
Message	One entry in the list sent to a model: a role (who is speaking) and its text. The list is the conversation so far.
Role	The label on a message. A system message holds standing instructions, a user message holds what the person said, an assistant message holds an earlier model reply.
Stateless call	A call that carries no memory of earlier calls. The model sees only the messages in this request, so your code must send the history again each time.
Finish reason	A field in the reply that says why the model stopped: it reached a natural end, it hit the length limit, or it is asking for a tool.
Usage	The token counts in a reply: how many tokens went in (the prompt) and how many came out (the completion).
Endpoint	The web address a program sends requests to. Hosted providers and local model servers each have one, and many accept the same chat format.
Schema	A written description of the shape a piece of data must have: which keys, which types, which allowed values and ranges.
Validation	Checking data against a schema and rejecting it when it does not fit, instead of passing it on.
Pydantic model	A Python class that describes a schema and checks data against it. Building one from bad data raises an error that says which field failed.
Function calling	A way for the model to ask for a function to be run: it replies with a function name and arguments, your code runs the function, and you send the result back. Also called tool calling.
Tool call	The model’s request to run one function, holding an id, the function name and the arguments written as JSON text.
Input and output tokens	The tokens you send (everything in the request, including tool descriptions) and the tokens the model writes back. Providers count and charge for the two separately.

Explanations

1. One request, one response

Here is the whole conversation a shop’s support helper has with a model, as a few lines of Python. Read it before reading any explanation:

from app.llm import chat, reply_text

messages = [
    {"role": "system", "content": "You are a support assistant for an online shop. Answer in one sentence."},
    {"role": "user", "content": "Can I return an unopened lamp?"},
]
reply = chat(messages, temperature=0.2)

print(reply_text(reply))                         # the words
print(reply.choices[0].finish_reason)            # why it stopped
print(reply.usage.prompt_tokens, reply.usage.completion_tokens)   # tokens in, tokens out

An API call is like handing a form across a post-office counter. The form has fixed boxes: who is writing, what they say, a few tick-boxes for settings. The clerk reads the form, writes the reply on a slip, and stapled to the slip is a receipt that says how much was used. In the same way, the messages are the form, settings such as temperature are the tick-boxes, and the response carries the text plus a receipt. A second picture: a kitchen order slip. The waiter writes the table, the dishes and any special requests; the kitchen sends back the plate and a bill.

Each message has a role. A system message holds standing instructions, user holds the person’s text, and assistant holds something the model said earlier. One source gap to know about: the current reference for one provider says that for its newest models a developer message takes the place of the older system message, so check which name your endpoint expects.

What comes back is an object with a list called choices. For one answer you read the first item, reply.choices[0].message.content, which is the text. Beside it sit two fields worth checking on each call:

• Finish reason. It says why the model stopped. stop means a natural end or a stop sequence you supplied, and length means the token limit you set was reached. A reply that ends mid-sentence with length is not a model mistake; you asked it to stop. Here is what that looks like, with an invented reply: the text reads Items can be returned within 30 days if they are unused and in their orig and the finish reason is length. The fix is to raise the limit you set, or to ask for a shorter answer. The name of the limit setting differs between endpoints; one provider’s reference lists max_completion_tokens and marks the older max_tokens as deprecated, so read the documentation of the endpoint you use.
• Usage. It holds prompt_tokens (tokens in) and completion_tokens (tokens out), plus a total. These are the numbers you use to see whether a prompt fits (Chapter 1) and what a call costs (section 5).

Settings. Temperature, from Chapter 1, is passed with the request. One provider’s reference describes it as a value between 0 and 2, where higher values make the output more random and lower values more focused. The range can differ on other endpoints, so treat the number as endpoint-specific.

The model remembers nothing between calls. Here is the failure first. A program sends “My name is Asha.” and gets a friendly greeting. Then it sends a second request that contains only “What is my name?” and the reply is “I don’t know your name.” Nothing broke. One provider’s guide says each text generation request is independent and stateless. The model sees only the messages in the request in front of it. The picture to hold: a clerk with no memory who reads whatever file you hand over at the counter, and who does not know you. If you want a multi-turn conversation, you hand over the earlier messages again each time, including the assistant’s earlier replies:

history = [{"role": "user", "content": "My name is Asha."},
           {"role": "assistant", "content": "Nice to meet you, Asha."},
           {"role": "user", "content": "What is my name?"}]

The cost of this design is that the request grows with every turn (section 5) and must still fit in the context window from Chapter 1.

When the call itself fails. The openai package raises openai.APIConnectionError when it cannot reach the endpoint (a network problem or a timeout) and a subclass of openai.APIStatusError when the endpoint answers with an error status such as 4xx or 5xx. All of them inherit from openai.APIError. Catch these, not raw network errors, so a failed call can be reported or retried on purpose.

2. Hosted and open models behind the same shape of API

Think of a wall socket and a plug. A lamp does not care which power station feeds the wall; it needs the plug to fit. A second picture: a phone charger cable that fits phones from several makers because they agreed on the connector. Many model providers, and many programs that serve open models on your own computer, accept the same chat request format. A hosted model is run by a provider: you reach it over the internet with a key, and you usually pay by the token. An open model is one whose weights you can download and run yourself on a model server, so a request can stay on your machine.

The openai Python package is written for the OpenAI-style format, and it lets you point at any endpoint that speaks it. Its documentation says you can set a base_url in the client or use the OPENAI_BASE_URL environment variable. One local model server’s guide shows a client pointing at an address on your own machine with an API key that is “required but ignored”: the package wants a key, but that server ignores it. This course uses three values, held in environment variables, so the code stays the same:

Variable	What it holds
LLM_BASE_URL	the address of the endpoint, hosted or local
LLM_API_KEY	the key the endpoint expects; for a local server, any placeholder text
LLM_MODEL	the model name that endpoint lists

The helper that every later chapter imports reads those three values once per call and passes everything else straight through:

import os
from openai import OpenAI


def get_client():
    return OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"])


def chat(messages, temperature=0.2, **settings):
    return get_client().chat.completions.create(
        model=os.environ["LLM_MODEL"], messages=messages,
        temperature=temperature, **settings)


def reply_text(reply):
    return reply.choices[0].message.content or ""

Save it as app/llm.py (an empty app/__init__.py beside it makes the folder importable). Set the three values in your terminal before you run anything (for example export LLM_MODEL=... on macOS or Linux). Keep the key out of files in your repository; the library’s own guide recommends keeping it out of source control.

Here is the failure to expect. You develop against a hosted endpoint, then switch to a local server, and a request option your code relied on is rejected or silently ignored. The local server’s own guide says it supports “a subset of the OpenAI API”. The format is shared; the features are not guaranteed to match. So test on the endpoint you will actually use, and do not make your program depend on a feature you have not checked. That is one reason section 3 validates the output in your own code instead of trusting the endpoint to enforce a shape.

3. JSON output a program can trust

Here are two replies a program really can receive when it asks a model for a support ticket as JSON. Both are invented but typical:

Reply A:  Sure! Here is the JSON you asked for: {"category": "delivery", "urgency": 2, "summary": "Parcel late"}
Reply B:  {"category": "delivery", "urgency": "high", "summary": "Parcel late"}

Reply A is not JSON: json.loads stops at the word “Sure”. Reply B is valid JSON but the program expects urgency to be a number from 1 to 3, and "high" will break whatever does arithmetic on it later. A person reads both without trouble. A program needs a promise about the shape.

A form with tick-boxes gives that promise; a blank page does not. A schema is such a form for data: it lists the keys, the type of each, and the allowed values. A second picture: a security check at an airport gate. Everyone is welcome, but nobody boards without the right pass. Validation is that check: data that fits the schema goes through, data that does not is stopped at the gate with a reason. Two more everyday cases show why the reason matters. A parcel depot sends a parcel back with a note saying the label has no postcode; a teacher hands a form back with the one blank field marked so the pupil fixes just that field. A checker that says what is wrong lets the sender fix it, which is why the error text goes back to the model for the retry.

In Python, Pydantic turns a class into a schema:

from typing import Literal
from pydantic import BaseModel, Field

class Ticket(BaseModel):
    category: Literal["billing", "delivery", "other"]
    urgency: int = Field(ge=1, le=3)
    summary: str = Field(min_length=1, max_length=80)

The Pydantic documentation says that building a model raises a ValidationError when data cannot be parsed into it, and that model_validate_json checks a JSON string directly. It can also write the schema out: model_json_schema returns the schema as a dictionary, which you can paste into a prompt so the model sees the same rules your code enforces.

The routine has four steps, and the last two are what make it dependable:

1. Ask for one JSON object and nothing else, naming the keys and the allowed values.
2. Cut out the object from the reply (from the first { to the last }), which removes a friendly lead-in.
3. Validate it with the model class. Reply B fails here with an error that names the field urgency.
4. Retry or stop. Send the model its own bad reply plus the error text and ask again, once. If it still fails, raise a clear error; do not pass bad data on.

Some providers can enforce a schema for you. One provider’s guide separates two features: JSON mode, which makes sure the output is valid JSON, and Structured Outputs, which it says is the one that ensures the output follows your supplied JSON Schema. The same guide says its Python library accepts Pydantic classes for this. That is useful when your endpoint offers it. But section 2 showed that endpoints differ, and a provider-side promise does not help when you switch endpoints. Your own validation travels with your code, so keep it even when the endpoint enforces a schema.

4. Function calling: the model asks, your code runs the function

A customer asks, “Where is order 1042?” A model alone cannot know. It might invent “It shipped yesterday”, which is fluent and wrong. The model has no connection to your order system, and it cannot run your code.

At a restaurant the waiter writes down your order and walks it to the kitchen; the waiter does not cook. In function calling, the model is the waiter: it writes down which function it wants and with what arguments. Your code is the kitchen: it runs the function. A second picture: a receptionist takes a message, “please check order 1042”, and passes it to a colleague who has access to the order system; the colleague reports back and the receptionist tells the customer.

One provider’s guide describes the flow in five steps, and the same shape is used by many endpoints:

1. Your code sends a request that includes the tools the model may call.
2. The model replies, not with text, but with a tool call.
3. Your code runs the function with the arguments from the tool call.
4. Your code sends a second request that includes the function’s result.
5. The model replies with the final text (or with more tool calls).

Each tool is described by a name, a description and its parameters written as a JSON Schema:

TOOLS = [{"type": "function", "function": {
    "name": "get_order_status",
    "description": "Look up the status of an order by its four-digit number.",
    "parameters": {"type": "object",
                   "properties": {"order_id": {"type": "string"}},
                   "required": ["order_id"]}}}]

The model decides from the name and the description whether to call it, so write the description for a reader who does not know your system, like the prompt advice in Chapter 1. The guide says a tool call arrives with an id and a function holding a name and JSON-encoded arguments. The arguments are text, so your code parses them (json.loads) and, as in section 3, checks them against a schema before running anything. Here is the failure to guard against: the model asks for get_order_status with {"order_id": "ten forty-two"}. If your function trusts the argument it will search for nonsense; if it validates, it can return an error message the model can read and correct.

Your function’s result goes back as a message with the role tool and the id of the call it answers:

messages.append({"role": "tool", "tool_call_id": call.id, "content": "shipped"})

Then you send the whole list again and the model writes the final sentence from the real status. Note what stayed in your hands: the model did not touch the order table. It asked, your code decided whether and how to run the function, and your code returned a result. That gap is where later chapters put limits and checks. If the model asks for several calls at once, run each one and answer each by id.

5. What a call costs: input and output tokens

A courier who bills by the weight you send and the weight you get back makes a good picture. So does a phone plan that counts messages sent and received. A call to a model works the same way: you pay for what goes in and what comes out, and the two are counted separately. One provider’s documentation says a request is priced from the total input tokens sent, including those in the tools description, and the output tokens generated. Each provider sets its own rates, so this chapter uses invented rates in “units”; read the real rates from your provider’s documentation when it matters.

Work one call by hand. Suppose a call sends 350 input tokens and receives 120 output tokens. The invented rates are 2 units per 1,000 input tokens and 6 units per 1,000 output tokens.

• Input: 350 / 1,000 x 2 = 0.70 units
• Output: 120 / 1,000 x 6 = 0.72 units
• Call: 0.70 + 0.72 = 1.42 units

A feature that answers 1,000 questions of this size uses 1,420 units. You read the two token counts from the usage field of each reply, so the cost can be logged on every call.

Now the part that surprises beginners. Remember that the model is stateless, so a chat sends the history again each turn. Suppose a system message of 100 tokens, and each turn adds a 40-token question and a 60-token reply. Turn 1 sends 100 + 40 = 140 input tokens. Turn 2 sends 100 + 100 + 40 = 240, turn 3 sends 340, turn 4 sends 440 and turn 5 sends 540. Five turns send 140 + 240 + 340 + 440 + 540 = 1,700 input tokens, but only 5 x 60 = 300 output tokens. At the invented rates that is 1,700 / 1,000 x 2 = 3.4 units in, 300 / 1,000 x 6 = 1.8 units out, 5.2 units in all. A version with no history would send 140 tokens five times (700) and cost 1.4 + 1.8 = 3.2 units, but it would forget everything. The history is a trade: memory costs input tokens, and the cost grows with each turn until you trim or summarise the old messages.

Three levers follow: shorten the standing instructions, cap the reply length, and trim the history. Each trades something away, so measure with usage before and after.

Three situations to recognise

1. A nightly job that files tickets crashes on json.loads because the model began its reply with “Sure! Here is the JSON”. Nothing checks the shape between the model and the job. The fix is the routine of section 3: cut out, validate, retry once, then stop with a clear error.
2. An order helper invents shipping dates for the orders it is asked about. The model was not given a way to look anything up. Function calling lets it ask, and your code runs get_order_status against the real table.
3. The bill for a chat feature is far above the estimate made from one question and one answer. Each turn resends the history, so input tokens grow with the length of the conversation. Read usage per call and trim the history.

What this means for the design you write:

If the situation is	Do this	Because
The program needs fields, not prose	Ask for JSON, then validate with a schema	The model’s reply is text, and only validation checks the shape
The model needs data or actions only your code has	Describe a function and let the model call it	The model asks; your code decides whether to run it
The chat needs memory	Send earlier messages again, and trim them as they grow	Each call is stateless and input tokens are counted
You may switch endpoints	Keep model name, key and address in environment values, and validate output yourself	Endpoints share a format but not every feature

Flashcards

Say the answer aloud before revealing it. Speaking is a different skill from reading; practise both.

1. What does one chat request contain?

   • A list of messages, each with a role and its text, plus settings such as temperature and a limit on the reply length. The response holds the text, a finish reason and the token counts.

2. Why must your code resend earlier messages in a chat?

   • Each call is stateless: the model sees only the messages in that request. Without the history it cannot know what was said before, so the code sends the earlier turns again, trimmed when they grow long.

3. What does a finish reason of length tell you?

   • The reply stopped because the token limit was reached, so it may end mid-sentence. Raise the limit or ask for a shorter answer.

4. What three values change when you move the same code from a hosted model to a local one?

   • The address of the endpoint, the key it expects and the model name. The request format stays the same, though a local server may support only a subset of the features.

5. Why is json.loads on a model reply risky?

   • The model may wrap the JSON in friendly text, or return valid JSON with the wrong type or an unexpected value. A program needs the shape checked before it uses the data.

6. What is validation, and what do you do when it fails?

   • It is checking data against a schema and rejecting what does not fit. On failure you can retry once by sending the model its bad reply with the error text, and if that fails too, raise a clear error.

7. What is the difference between JSON mode and Structured Outputs in one provider’s guide?

   • JSON mode makes the output valid JSON; Structured Outputs also ensures the output follows your supplied schema. Keep your own validation too, because an endpoint may offer neither.

8. In function calling, who runs the function?

   • Your code. The model replies with a tool call holding the function name and arguments. Your code runs the function and sends the result back in a tool message.

9. Why validate the arguments of a tool call?

   • They arrive as JSON text written by a model, and may be missing or wrong. Checking them first lets you send back a clear error instead of running the function on nonsense.

10. What does a call cost, and where do you read the numbers?

    • Input tokens times the input rate plus output tokens times the output rate. The counts come from the usage field of the reply, and rates come from the provider’s documentation.

11. Why does a long chat get more expensive each turn?

    • Each request resends the whole history, so input tokens grow with every turn while each new reply stays about the same size. Trim or summarise old messages to control it.

Model answer

The core question this unit answers: “How do you get an answer from a hosted or local model that a Python program can read and act on without breaking?”

Five beats a good answer covers; hit all five, in this order:

1. Request — messages and settings, with the history resent each time
2. Reply — text, finish reason and token counts
3. Structure — ask for JSON, validate against a schema, retry once
4. Function — the model asks, my code runs it
5. Cost — input and output tokens, counted separately

The full answer, said aloud:

(Request) “A call sends a list of messages and some settings to an endpoint, and because the model is stateless I send the earlier messages again each time. The same code works for a hosted or a local model once the address, key and model name are set.

(Reply) What comes back is the text, a finish reason that tells me if it was cut off, and the token counts.

(Structure) Because a program cannot use a chatty paragraph, I ask for JSON, cut it out of the reply, validate it against a schema, and retry once with the error before I stop with a clear failure.

(Function) If the model needs data or actions only my code has, I describe a function; the model replies with a tool call, my code validates the arguments and runs it, and I send the result back.

(Cost) Every call is measured in input and output tokens, counted separately, so I read the usage field and watch how a growing history raises the input.“

Skip any of these five and the answer is incomplete. An interviewer or examiner who hears only “you send a prompt and get JSON back” will ask what happens when the JSON is wrong, and who runs the function. Practise until all five beats come out in order without notes.

Why this matters

On the job: many AI features are programs that call a model and then do something with the reply. In our view, the difference between a demo and a feature often lies in what happens when the reply is not what you expected, and a validated schema is the first answer to that.

In interviews and reviews: “how do you get reliable JSON from a model?” and “how does function calling work?” are common questions for an AI engineering role. The five-beat answer above is built for them.

As a builder: the helper chat() that you read here is the one every later chapter calls, so the shape of the request and the habit of reading usage will come back in each of them.

After this unit

Hands-on project. Three tasks: (1) work out the cost of three invented calls from their usage records and a rate table, in a small Python program; (2) write a schema for a support ticket and a checker that accepts or rejects six invented model replies, saying why; (3) optionally, run your schema and the chat() helper against a real endpoint. The full brief is in the project file.

What’s next. Chapter 3 gives the model passages from your own documents, found by meaning, so it can answer from them and name the pages. It sends every question to the model through the chat() helper from this chapter. Keep llm.py, schemas.py and your checker: the next chapter builds on them.

Hands-on Project — Chapter 2: Cost a Call, Check a Reply

Objective

Write two small Python programs that make a model call measurable and safe to use (one works out what calls cost from their token counts, the other accepts or rejects model replies against a schema), then optionally run both against a real model endpoint.

What you need: Python 3, a text editor, and pip install pydantic for Task 2. Tasks 1 and 2 need no model, no account, no API key and cost nothing. Task 3 is optional and needs a model endpoint: a model server running on your own computer, or a hosted provider, which needs an account and key and may charge by the token, so check its rates first. Every token count, rate and reply in this brief is invented for the exercise; none comes from a real model.

You write every line yourself. This brief gives the facts and the checks; the finished programs and the right verdicts are not in it. Work through the tasks in order and check each output against its checks before moving on.

---

Task 1 — Cost three calls and one chat

Create cost.py. The invented rates are 3 units per 1,000 input tokens and 9 units per 1,000 output tokens.

Part A. Three usage records, as a model would report them:

Call	Input tokens	Output tokens
A	420	90
B	1,250	310
C	60	15

Write a function call_cost(input_tokens, output_tokens) that returns the cost in units. Print one row per call with its input cost, output cost and call cost, then a total row.

Part B. A five-turn chat resends its history every turn. The system message is 80 tokens. Each turn adds a question of 30 tokens and a reply of 50 tokens, and the question and reply of a turn are part of the input of every later turn. Print a table with one row per turn: the input tokens sent, the output tokens received and the cost of that turn. Then print the chat’s total cost with the history resent, and its total cost if each turn had sent only the system message and the new question.

Scope: three records and five turns, not more. Stop once both tables are printed and the checks below hold.

Checks:

• Part A has 3 call rows and 1 total row, and the total equals the sum of the three call costs.
• Each call cost equals the input cost plus the output cost.
• In Part B the input tokens sent grow by the same amount from one turn to the next.
• The total with history is larger than the total without, and your output says by how much.

Output: the printed tables (copy them into cost-output.txt) and your cost.py.

---

Task 2 — Check six model replies against a schema

Create check_replies.py. A program asked a model for a parcel update as JSON. The schema it needs is:

• order_id: text of exactly four digits
• status: one of packed, shipped, delivered, returned
• days_late: a whole number from 0 to 30
• note: optional text of at most 60 characters

Describe it as a Pydantic class named Parcel. Then write a function check(reply) that does the routine from the notes: cut the JSON object out of the reply (from the first { to the last }), validate it with the class, and return either the accepted object or a reason for rejection.

Run it on these six invented replies and print one line per reply, starting with ACCEPT or REJECT:

1. {"order_id": "1042", "status": "shipped", "days_late": 0}
2. Sure! Here you go: {"order_id": "2210", "status": "delivered", "days_late": 3, "note": "left with neighbour"}
3. {"order_id": "77", "status": "shipped", "days_late": 1}
4. {"order_id": "3055", "status": "lost", "days_late": 2}
5. The parcel is probably on its way.
6. {"order_id": "4120", "status": "packed", "days_late": "two"}

Scope: one class, one function and six replies. Stop there.

Checks:

• Every reply gets exactly one verdict line.
• Every rejected reply prints a reason that names the field that failed, or says that no JSON object was found.
• Every accepted reply prints the parsed values.
• Your Parcel class contains no if statements: the rules live in the field types and constraints.

Output: the six verdict lines (copy them into verdicts.txt) and your check_replies.py.

---

Task 3 — Run both against a real endpoint (optional)

Save the chat() helper from the notes as app/llm.py, set LLM_BASE_URL, LLM_API_KEY and LLM_MODEL in your terminal, and keep the key out of every file. Send one request that asks the model to answer with a single JSON object for the sentence “Order 1042 shipped two days late”, using the four keys of Task 2, with the allowed values written into the prompt. Pass the reply through your check() from Task 2, and read the token counts from the reply’s usage field.

Scope: one request. If the reply is rejected, you may send one retry that includes the error text, and then stop.

Checks:

• The environment values are read by the helper; no key or address appears in any file you keep.
• Your output shows the reply text, the finish reason, the verdict from check(), and the cost of the call from your call_cost function.
• If you needed a retry, the output shows both the first verdict and the second.

Output: a file live-run.txt with the reply text, the finish reason, the verdict, the token counts and the cost.

Keep llm.py, check_replies.py and cost.py: the next chapter sends every question to the model through the chat() helper.