Chapter 5 — Machine-Usable Output

Course: Getting Started with AI

Prerequisites: Chapters 1 to 4, with the ideas re-defined in the notes before they are used: a response ends with a stop reason that says why generation stopped, a prompt states an output contract and shows examples of it, and every token in the window is paid for and budgeted. You can read and run a short Python script. No account with a model provider and no API key are needed.

What the reader can do after this chapter:

• Write a schema for a model’s reply, with its required fields, types, allowed values and a rule against extra fields, and say which of those a validator checks.
• Write a validator that reads a reply as text and returns the first problem it finds as a named code, and use it to separate a malformed reply from a valid one.
• Decide what to do with a bad reply from its stop reason and its error code: retry with a larger limit, retry within a bounded number of attempts, treat a refusal as its own outcome, or fail.

Core question this chapter answers: When a program, not a person, reads a model’s reply, how do you define the shape that reply must have, make the model produce it, check it before anything uses it, and decide what to do when it is wrong?

Connections:

• Builds on: Chapter 4’s budgeted request: the reply that has to fit in its output reserve, and the request’s system message, examples and user message.
• Leads into: Chapter 6, which looks at how the same request is sent to more than one provider and model, and why scattering provider-specific calls through a codebase becomes a trap.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for an answer in an exact format, and each one is marked by a rule that the format printed with the question lets you check.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines, and Task 3 also from the files it asks you to bring forward. Every accepted value, or the rule that decides it, follows from the data, formulas and rules printed in the project, so you can check your own work before you submit it.

Getting Started with AI — Chapter 5: Machine-Usable Output

Before you start

Nothing to install for the reading. For the project you need Python 3 and its standard library only; no package is needed. You do not need an account with any model provider, an API key or any payment.

Keep one question in mind while you read: if a program reads this reply instead of a person, what exactly would make it crash, and where would I have caught that? By the end of the chapter you should be able to define the shape a reply must have, check it before anything uses it, and say what happens when it is wrong.

Summary

A person can forgive a reply that is a little off; a program cannot. When code reads a model’s output, the output needs a defined shape, usually a JSON schema that names the fields, their types, their allowed values and whether extra fields are tolerated. Providers offer ways to push the model toward that shape, from asking in the prompt, to a mode that guarantees valid JSON, to schema-constrained generation, whose documentation still lists cases where output may not match, such as a refusal or a response cut off at its token limit. So the reply is validated at the boundary, in your code, before anything else touches it, and a failed check leads to a decided outcome: a retry, a larger limit, or an error that is raised rather than swallowed. A reply that passes every check can still be wrong, which is why validation is the first test of a reply and not the last.

Key concepts

Term	Definition
Schema	A written description of the shape of a piece of data: which fields exist, their types, which are required and what values they may take. JSON Schema is the standard vocabulary for describing JSON this way.
Required field	A field that must be present. In JSON Schema, an object is valid against required if every name in the list is a property of the object.
Enum	A closed list of allowed values for a field, such as bug, feature, question.
additionalProperties	The schema setting that says whether keys not listed in the schema are allowed. Set to false, extra keys make the object invalid.
JSON mode	A provider setting that makes the model produce syntactically valid JSON, without promising that it matches any particular schema.
Schema-constrained generation	A provider feature in which you send a schema and the model’s output is constrained to follow it. One provider’s documentation describes it as working through constrained decoding.
Validation	Checking data against rules before using it: it parses, has the right fields, the right types, allowed values and passes any further rules.
Boundary	The point where data crosses from a component you do not control, such as a model, into your own code. Checks belong here.
Stop reason	The field in a response that says why generation ended, such as finishing normally, reaching the token limit or refusing.
Refusal	A response in which the model declines a request, typically for safety reasons. Its text need not match your schema.
Bounded retry	Trying again after a failure, up to a fixed number of attempts, and then stopping with an error.

Explanations

The failure, before the explanation

A service asks a model to classify a support ticket and return a small JSON object, and a program then reads it. After launch the service receives these replies, among others, one per ticket:

1. Sure! Here is the JSON: {"label": "bug", "confidence": 0.9, "reason": "Crash."}
2. {"label": "bug", "confidence": 0.9, "reason": "Crash." (the closing brace never arrives)
3. {"label": "Bug", "confidence": 0.9, "reason": "Crash."}
4. {"label": "bug", "confidence": "high", "reason": "Crash."}
5. {"label": "bug", "confidence": 0.9}
6. {"label": "bug", "confidence": 0.9, "reason": "Crash.", "notes": "n/a"}

Each one breaks the program differently. The first is not JSON, because of the sentence in front of it. The second is not JSON because it stops partway. The third parses but its label is not one the program knows. The fourth has a string where a number belongs. The fifth lacks a field. The sixth carries a field nobody asked for, which a strict consumer may reject. None of these examples is a measurement of a real system; they are the kinds of defect the rest of the chapter gives names to, so that each can be caught on purpose.

Defining the shape: a schema

Start with the shape the program needs, written as a schema. Here is one for the replies above:

{
  "type": "object",
  "properties": {
    "label": {"type": "string", "enum": ["bug", "feature", "question"]},
    "confidence": {"type": "number", "minimum": 0, "maximum": 1},
    "reason": {"type": "string", "minLength": 1}
  },
  "required": ["label", "confidence", "reason"],
  "additionalProperties": false
}

Read it as a contract in four parts. type says the reply is an object. properties says what each field may be. required lists the fields that must be there; the JSON Schema specification says an object “is valid against” required “if every item in the array is the name of a property in the instance” (JSON Schema, Validation: A Vocabulary for Structural Validation of JSON). And additionalProperties: false rules out any key not listed. The six replies above fail it in turn: number 3 on the enum, number 4 on the type, number 5 on required, number 6 on additionalProperties. The first two fail earlier, before there is any object to check.

A second way to see it: a schema is a function signature for data. A signature says a function takes an integer and a string and returns a boolean, and a caller that passes anything else is wrong before the function body runs. The schema says the same about a reply, and the validator is the check that runs before the body of your program does.

Code-first teams often write the shape as a type and let a library produce the schema. One provider’s documentation shows a Pydantic model passed as the output type, with the SDK turning it into a schema, sending it, validating the response and returning the parsed object (Anthropic, Structured outputs); another provider’s libraries accept pydantic.BaseModel for the same purpose (OpenAI, Structured model outputs). Pydantic’s own documentation describes validation as instantiating a model that adheres to the declared types and constraints, and says a ValidationError is raised when data cannot be parsed into a model instance (Pydantic, Models). The advice from one provider on keeping the two from drifting apart: use native SDK schema helpers where available, or add checks that flag when either the schema or the data objects are edited (OpenAI, Structured model outputs).

Choosing what to put in a schema is a design act:

Choice	Use it when	The risk
An enum for a closed set	The program branches on the value	A model-produced value that differs only in capitalization, which one provider documents (below)
Required fields	The program cannot continue without the field	The model has to produce something even when the input has no answer, so give it an escape value
additionalProperties: false	A strict consumer rejects unknown keys	A provider’s feature can require it, so check the provider’s rules
A short reason field	A person will read the reply later	It costs output tokens on every call (Chapter 2)

Getting the model to produce it: three levels

Now the other side: how to make the model follow the shape. There are three levels, in rising strength.

Ask, and show. Say the exact shape in the system message and show examples that match it, as Chapter 3 did. This costs nothing beyond tokens. It cannot guarantee anything: the model can still add a sentence in front or end early. One provider’s documentation lists the failures that can remain even with careful prompting: parsing errors from invalid JSON syntax, missing required fields, inconsistent data types and schema violations requiring error handling and retries (Anthropic, Structured outputs).

JSON mode. A mode that makes the output valid JSON. The OpenAI documentation says that “while both ensure valid JSON is produced, only Structured Outputs ensure schema adherence”, and that its JSON mode does not adhere to a schema (OpenAI, Structured model outputs). Valid JSON addresses the first failure above, the prose in front, but not the second: a reply cut off at its token limit is not complete JSON either, and the OpenAI documentation’s own JSON-mode example checks for a length finish reason that means incomplete JSON. Replies 3 to 6 are already valid JSON.

Schema-constrained generation. You send the schema, and the provider constrains generation to follow it. One provider says its structured outputs “guarantee schema-compliant responses through constrained decoding”, with the schema sent in output_config.format (Anthropic, Structured outputs); the other says its Structured Outputs feature “ensures the model will always generate responses that adhere to your supplied JSON Schema” (OpenAI, Structured model outputs). Both pages also say where that stops, which the next section takes up. Both describe limits on which parts of JSON Schema they support. The first lists numeric limits such as minimum and maximum and string limits such as minLength as unsupported, and its SDK helpers remove them from the schema that is sent while a helper that validates responses still checks them in your code (Anthropic, Structured outputs).

There are practical costs. The first provider says the first request with a new schema has extra latency while its grammar compiles, with compiled grammars cached for 24 hours from last use; it also says the model automatically receives an additional system prompt explaining the output format, so input token counts are slightly higher, and that changing the format parameter invalidates a prompt cache for that conversation thread (Anthropic, Structured outputs). The second says the first request with any schema has additional latency, and later requests with the same schema do not (OpenAI, Structured model outputs). Chapter 2’s rules apply: a schema is part of the cost and the cache story.

Which level to choose depends on what breaks you: if a human reads the output and an occasional slip is tolerable, asking and showing may do. If a program consumes it, use the strongest level the provider offers for your model and still validate. A reader who stops at “the provider guarantees the schema” has stopped one step early.

Validating at the boundary

A reply from a model is data from outside your code. The habit that matters is to check it where it enters, and the check has layers, each of which catches something the previous layer lets through:

Layer	Question	Catches
Parse	Is it JSON at all?	Replies 1 and 2: prose around it, truncation
Shape	Is it an object with the right fields and types?	Replies 4, 5 and 6
Values	Are the values allowed, in range, non-empty?	Reply 3, and a confidence of 1.4
Meaning	Is it correct for this input?	Nothing above; see the next section

In Python the first layer is json.loads, which raises JSONDecodeError “when the data being deserialized is not a valid JSON document” (Python, json). Pydantic’s model_validate_json() validates JSON data against a model. Pydantic’s documentation draws a line worth remembering: it “guarantees the types and constraints of the output, not the input data”, and a ValidationError is what it raises when parsing into the model fails (Pydantic, Models).

Why check again when the provider guarantees the schema? The provider’s own pages give the reasons. The first says that while structured outputs “guarantee schema compliance in most cases”, there are scenarios where the output may not match: a refusal, where “the refusal message takes precedence over schema constraints”; and the token limit, where “the output may be incomplete and not match your schema” (Anthropic, Structured outputs). It adds a third, easy to miss: the capitalization of enum values is not guaranteed, so a value can differ from the schema only in capitalization, with no error and no special stop reason, and the advice is to compare enum values case-insensitively and avoid enum values that differ only in capitalization. The second provider says the model “might not generate a valid response that matches the provided JSON schema” in the case of a refusal or an incomplete response at a max tokens limit (OpenAI, Structured model outputs), and its refusals appear in a separate refusal field because they do not necessarily follow the schema.

So the order of the checks matters: look at the stop reason and any refusal field first, then parse, then validate shape and values. A reply that stopped at the token limit is not a malformed reply to be patched; it is an unfinished one.

Schema-valid is not the same as correct

A reply can pass every layer and still be wrong. The OpenAI guide says it directly: “Structured Outputs can still contain mistakes. If you see mistakes, try adjusting your instructions, providing examples in the system instructions, or splitting tasks into simpler subtasks.” It also warns about input that does not fit the task: “The model will always try to adhere to the provided schema, which can result in hallucinations if the input is completely unrelated to the schema”, and suggests telling the model in the prompt what to return, such as empty parameters or a specific sentence, if the input is incompatible with the task (OpenAI, Structured model outputs).

Here is what that looks like for the classifier. A ticket arrives that contains no complaint, request or question, only a marketing email forwarded by mistake. A schema that requires a label from three values gives the model no way to say “this is none of them”, so any reply that fits the schema is a guess. The fix is in the contract, not the validator: add a value such as other to the enum, or an explicit “not applicable” field, and say in the rules when to use it.

A second way to see it: a form with a required “Reason for visit” drop-down will be filled in by everyone who uses it, including people with none of the listed reasons. The form is complete and the data is wrong. A validator, like a form, can check that a box was ticked, not that it should have been.

When the reply is wrong: a decision procedure

Last, what to do when a check fails. Here is a procedure that this chapter’s project implements. It is the chapter’s exercise policy, assembled from the documented edge cases above; it is not a rule that any provider publishes.

1. Refusal? A refusal is its own outcome. Do not retry the same request hoping for valid JSON; pass it to whatever handles refusals.
2. Stopped at the token limit? The reply is unfinished. The provider’s page says to retry with a higher max_tokens value (Anthropic, Structured outputs); you can also shorten what you ask for. Do not parse the fragment.
3. Otherwise parse and validate. If valid, use it.
4. If invalid, retry a bounded number of times. Each retry costs tokens and time (Chapter 2). Stop after a fixed number of attempts and fail with the last error code, loudly. A loop with no bound is a bill with no ceiling.
5. Log the failure and its code. The count of each code over time tells you what to fix: a rise in enum failures points at the prompt or the schema; a rise in truncation points at the output limit.

What you see	Which step	What to do
Reply cut off, stop reason max_tokens	2	Raise the limit or ask for less; never parse the fragment
Stop reason refusal, output is plain text	1	Handle as a refusal; do not count it as malformed JSON
Valid JSON, label is Bug	3 and 4	Compare enum values case-insensitively if the provider says casing is not guaranteed; otherwise retry
Valid, schema-conforming, wrong	Not a validator problem	Fix the contract or the prompt; add an escape value

Three situations, and what they mean for your design

Before the table, three situations:

1. A pipeline reads a model’s JSON and stops with a parse error on about one reply in a hundred. Every failing reply ends mid-field.
2. A team turns on schema-constrained output and removes its validation code. Later, a field that should be between 0 and 1 arrives as 7 in some replies.
3. A classifier schema has three labels. Tickets about billing, which the labels do not cover, are being assigned one of the three confidently.

What you see	Which part of the design	What to do
Parse errors on replies that end mid-field (situation 1)	Output limit	Check the stop reason; raise max_tokens or shorten the reply; treat a length stop as a failure
A value outside its range (situation 2)	Boundary validation	Keep validating ranges in code: a provider can leave a constraint out of the schema it enforces
A confident wrong label (situation 3)	The contract	Add an other value or an explicit not-applicable field; test with out-of-scope inputs

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

Why does a program need a defined output shape when a person does not?

A person forgives a reply that is slightly off. A program fails on a missing field, a wrong type, an unknown value or text around the JSON, so the shape has to be written down and checked.

What does a schema describe, and what does `required` mean in JSON Schema?

It describes which fields exist, their types, which are required and what values they may take. An object is valid against required if every item in the list is the name of one of its properties.

What is the difference between JSON mode and schema-constrained output?

JSON mode makes the output valid JSON. Schema-constrained output also makes it adhere to the schema you send, which JSON mode does not.

If a provider guarantees schema compliance, why validate in your own code?

The provider’s own pages list cases where output may not match: a refusal, a response cut off at the token limit, and enum values whose capitalization is not guaranteed. Some constraints are also left out of the schema the provider enforces, so only your code checks them.

What are the layers of a boundary check, in order?

Parse (is it JSON), shape (object, fields, types), values (allowed, in range, non-empty), then meaning (is it correct for this input). The first three can be automated; the last needs tests on real inputs.

A reply fails to parse and its stop reason is `max_tokens`. What do you do?

Treat it as unfinished, not malformed. Retry with a higher limit or ask for a shorter reply. Do not try to repair and use the fragment.

How should code treat a refusal when structured output is on?

As its own outcome, checked before parsing. The refusal’s text need not match the schema, and retrying the same request to get valid JSON is not the right response.

What does one provider say about enum values and capitalization?

Capitalization of string enum values is not guaranteed: the model may return a value that differs from the schema only in capitalization, with no error. Compare case-insensitively and avoid enum values that differ only in capitalization.

Can a reply that passes the schema still be wrong?

Yes. The OpenAI guide says structured outputs can still contain mistakes, and that unrelated input can lead to hallucinated values that fit the schema. Fix the contract, for example with an other value.

Why bound retries?

Each retry costs tokens and time. Without a bound, a systematic failure becomes an open-ended cost, so stop after a fixed number of attempts and fail with the last error code.

What do structured outputs add to cost and latency on one provider?

The first request with a new schema has extra latency while the grammar compiles, the model gets an extra system prompt describing the format so input tokens are slightly higher, and changing the format invalidates a prompt cache for that conversation thread.

Model answer

The question: “A downstream service parses our model’s replies, and it breaks on a small share of them. How do you make the output machine-usable?” This is the question a lead engineer asks after the incident above. A strong answer hits five beats, in this order:

1. Contract — a schema: fields, types, required fields, allowed values, no extra fields.
2. Constrain — ask and show examples, and use JSON mode or schema-constrained output where the provider offers it for the model in use.
3. Validate — at the boundary, in order: stop reason, parse, shape, values.
4. Handle — a refusal is its own outcome, a length stop is not parsed, other failures get a bounded retry and then a loud error, with the code logged.
5. Verify meaning — a valid reply can still be wrong: test with real and out-of-scope inputs, and give the contract an escape value.

“First I’d write the reply’s shape as a schema: the fields, their types, which are required, the allowed values for any closed set, and no extra keys. Then I’d make the model follow it. I’d say the shape in the prompt and show examples, and where the provider offers schema-constrained output for our model I’d turn it on, because JSON mode only gives valid JSON, not the schema. But I’d still validate in code at the boundary, because the providers’ own documentation lists cases where output may not match: refusals, a cut-off at the token limit and enum capitalization. I’d check the stop reason first, then parse, then check the shape and values. A length stop I’d treat as unfinished and retry with a higher limit, a refusal I’d handle on its own, and anything else would get a bounded retry and then an error with its code logged. And I’d remember that a valid reply can still be wrong, so I’d test with inputs that fit none of the categories and give the schema an ‘other’ value for them.”

Missing a beat is a failure state. Without contract nothing says what correct looks like. Without constrain every reply is a roll of the dice. Without validate a bad reply reaches code that trusts it. Without handle a failure has no defined outcome. Without verify meaning the system passes every check and is still wrong — and that is the beat reviewers most often find missing.

Why this matters

• Reliability. A pipeline that parses model output is only as reliable as its handling of the replies that do not parse. Named failure codes turn “it crashed sometimes” into counts you can act on.
• Cost. Retries, a longer output limit, a schema’s extra prompt tokens and a schema change that invalidates a cache are all costs of the choices in this chapter (Chapter 2).
• Debugging. A log of failure codes tells you whether the prompt, the schema, the output limit or the provider is the one at fault.
• Trust. Systems that act on model output, by writing to a database or calling another service, need the boundary check before the action, not after.

After this chapter

The project for this chapter is Validate a Model’s Reply (project.md). You will write a validator that names the first problem in a reply, write the decision procedure that chooses between accepting, retrying, refusing and failing, and finish by writing a schema and sample replies for the request you have carried through the course.

Chapter 6 starts from the same request. It asks what happens when that request has to run against more than one provider or model: how to keep one interface, pin versions, migrate between models, and why provider-specific calls scattered through a codebase become a trap.

Hands-on Project — Validate a Model’s Reply

Objective

Write a validator that reads a model’s reply as text and names the first problem in it, write the decision procedure that chooses between accepting, retrying, refusing and failing, and finish by writing a schema and sample replies for the request you have carried through the course, so that a program can tell a usable reply from an unusable one before anything acts on it.

What this project needs: Python 3 and its standard library only (json; see Python, json, for json.loads and the JSONDecodeError it raises). No task needs a package, an account with a model provider, an API key or any payment, and no task calls a model. All three tasks are required.

Your evidence file. Keep one plain-text file, chapter5-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print.

Task 1 — Write the validator

Here is a schema, written as a Python dictionary. It is the schema from the notes. Copy it exactly:

SCHEMA = {
    "type": "object",
    "properties": {
        "label": {"type": "string", "enum": ["bug", "feature", "question"]},
        "confidence": {"type": "number", "minimum": 0, "maximum": 1},
        "reason": {"type": "string", "minLength": 1},
    },
    "required": ["label", "confidence", "reason"],
    "additionalProperties": False,
}

Write validate(text, schema), which returns one string: "ok" or the first problem found, as a code. Use json.loads for parsing. The checks run in this order, and the function returns as soon as one fails:

1. Parse. If json.loads(text) raises an error: not_json.
2. Object. If the parsed value is not a dictionary: not_object.
3. Fields, in the order they appear in schema["properties"]. For each field:
   • if it is absent and its name is in schema["required"]: missing:<field>; if it is absent and not required, move to the next field;
   • if it is present but its value does not have the declared type: type:<field>. The types are string (a Python str), number (a Python int or float), integer (a Python int) and boolean (a Python bool). A boolean is never a number or an integer, although the Python documentation says the bool class is a subclass of int;
   • if the field has an enum and the value is not in it: enum:<field>. Comparison is exact, including capitals;
   • if the value is below minimum or above maximum: range:<field>. The limits themselves are allowed;
   • if the value is shorter than minLength characters: length:<field>. Do not strip anything.
4. Extra keys. If schema["additionalProperties"] is False and the object has a key that is not in schema["properties"], return extra:<key> for the first such key in the order it appears.
5. Otherwise return ok.

Run it on these twelve replies, in this order, and print the result for each. Copy the strings exactly:

REPLIES = [
    '{"label": "bug", "confidence": 0.92, "reason": "Export fails."}',
    'Sure! {"label": "bug", "confidence": 0.9, "reason": "Crash."}',
    '["bug", 0.9, "Crash."]',
    '{"label": "bug", "confidence": 0.9}',
    '{"label": "Bug", "confidence": 0.9, "reason": "Crash."}',
    '{"label": "bug", "confidence": "high", "reason": "Crash."}',
    '{"label": "bug", "confidence": 1.4, "reason": "Crash."}',
    '{"label": "feature", "confidence": 0.7, "reason": ""}',
    '{"label": "question", "confidence": 0.8, "reason": "Asks how.", "notes": "n/a"}',
    '{"label": "bug", "confidence": true, "reason": "Crash."}',
    '{"label": "bug", "confidence": 0.9, "reas',
    '{"label": "bug", "confidence": 1, "reason": "Edge of range."}',
]

Add to your evidence file, one line per reply, numbered from 1 in the order above, and one count:

t1_r1: <code for the first reply>
t1_r2: <code for the second reply>
...
t1_r12: <code for the twelfth reply>
t1_ok_count: <how many of the twelve returned ok>

Output: one Python file with one function, and thirteen evidence lines. Stop there: one function and no more than the five checks above; you are not writing a schema library.

Task 2 — Decide what to do with each response

A real call returns a stop reason as well as text: Anthropic’s Structured outputs names stop_reason values refusal and max_tokens, and OpenAI’s Structured model outputs shows a finish_reason of length. This project uses the Anthropic names. Add to your Task 1 file a second function, handle(attempts, max_attempts, schema), which calls validate. attempts is a list of (stop_reason, text) pairs in the order the attempts would arrive; it stands in for repeated calls, so no call is made. Go through the attempts in order, counting each one you use, and stop at the first rule that fires:

1. If the stop reason is refusal: return ("refused", n), where n is the number of attempts used so far. A refusal is not retried.
2. If the stop reason is max_tokens: the reply is unfinished. Do not parse it. Remember truncated as the last problem and treat the attempt as failed.
3. Otherwise run validate(text, schema). If it returns ok: return ("ok", n). If not, remember the code as the last problem.
4. A failed attempt moves on to the next one, but you may use at most max_attempts attempts. If the max_attempts-th attempt failed, return ("failed:" + last_problem, max_attempts). Attempts after that are never looked at. If the list runs out before max_attempts with every attempt failed, return ("failed:" + last_problem, n).

This is an exercise policy that applies the edge cases the notes describe; it is not a rule that any provider publishes.

Use max_attempts = 3 and the SCHEMA and REPLIES from Task 1, referring to REPLIES by position (REPLIES[0] is the first). Run these five cases and print each result:

CASES = [
    [("end_turn", REPLIES[1]), ("end_turn", REPLIES[0])],
    [("max_tokens", REPLIES[10]), ("max_tokens", REPLIES[10]), ("end_turn", REPLIES[3])],
    [("refusal", "I can't help with that."), ("end_turn", REPLIES[0])],
    [("end_turn", REPLIES[4]), ("end_turn", REPLIES[11])],
    [("end_turn", REPLIES[1]), ("max_tokens", REPLIES[10]), ("end_turn", REPLIES[8]), ("end_turn", REPLIES[0])],
]

Add to your evidence file. Write each result as the outcome, a slash, and the attempts used, for example ok/1:

t2_c1: <outcome/attempts>
t2_c2: <outcome/attempts>
t2_c3: <outcome/attempts>
t2_c4: <outcome/attempts>
t2_c5: <outcome/attempts>
t2_total_attempts: <the sum of the attempts used over the five cases>

Output: the same Python file as Task 1, now with its second function, and six evidence lines.

Task 3 — A schema for your own request

Use the request you have carried since Chapter 2: its instruction, its input and the reply it should produce. If you do not have one, use a stand-in: a request that extracts a person’s name and age from one sentence. Write two files:

• schema.json — a JSON file with the same shape as SCHEMA: type is object; properties has at least three fields, each with a type of string, number, integer or boolean; at least one property has an enum, or a minimum or maximum; required is a non-empty list of names that are all in properties; additionalProperties is false.
• replies.json — a JSON list of exactly six strings, each a possible reply to your request, written by hand. Exactly three must be valid under your schema. The other three must be invalid, and between them they must show at least three different kinds of failure, where the kind is the part of the code before the colon (not_json, not_object, missing, type, enum, range, length or extra).

Run your Task 1 validator on all six replies with schema.json, loaded with json.load. Print the code for each, the number of valid and invalid replies, and the sorted list of distinct failure kinds. Fix any reply that is not what you intended.

Add to your evidence file. Write the failure kinds sorted alphabetically, comma-separated, no spaces:

t3_property_count: <number of properties in schema.json>
t3_required_count: <number of names in required>
t3_valid_count: <replies your validator accepts, which must be 3>
t3_invalid_count: <replies your validator rejects, which must be 3>
t3_failure_kinds: <distinct kinds among the three, sorted>
t3_distinct_failure_kinds: <how many kinds>

Output: two files and six evidence lines. Submit the two files with the evidence file; a reference validator that follows the Task 1 rules is run on them.

What carries forward

Keep schema.json, replies.json, your validator and chapter5-evidence.txt. The next chapter opens with the same request and asks what it takes to send it to more than one provider and model without rewriting the code that builds it, reads its reply or validates it.