Session 7 of 11
Human-in-the-Loop Approval Workflows
Overview
Chapter 7 — Human-in-the-Loop Approval Workflows
Course: Building AI Agents
Prerequisites: You can read and run a short Python script. From the previous chapter you need the idea that an
agent holds a set of permissions, each written resource:verb, and that its grant can be trimmed to what its task
needs (the agent-scopes.tsv that chapter’s project writes). Approval gate, reversible action, fingerprint,
approval lifetime, checkpoint, idempotent and idempotency key are defined in the notes before they are used.
What the reader can do after this chapter:
- Decide, from a stated rule, which of an agent’s pending actions must wait for a person and which may run on their own.
- Check an execution against its approval for the four ways it can fail: no approval, the wrong action, changed parameters, and an expired approval.
- Replay a sequence of retries against an execution that carries a key, and count the real effects it produces.
Core question this chapter answers: Where in an agent’s path must a human sit, and how do you build that checkpoint so that what the person approved is what runs, once, even after retries and restarts?
Connections:
- Builds on: the previous chapter’s trimmed permissions. It opens with the permissions in
agent-scopes.tsvthat create, change or remove something, and asks which uses of them should wait for a person. - Leads into: Chapter 8, which asks where the approved actions run and how far a single compromised
environment can reach. It opens with the effects your
executed.tsvrecords.
Your answers are scored
This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question.
- Nothing is judged by reading it for style. An answer either matches the rule or it does not.
- All three project tasks are scored from an evidence file of
key: valuelines and one tab-separated file. Every accepted value follows from what the task prints.
Notes
Building AI Agents — Chapter 7: Human-in-the-Loop Approval Workflows
Before you start
Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library. You do not need an account with any cloud or model provider or an API key: every action, approval and retry the project uses is printed in the project file.
Keep one question in mind while you read: when a person clicks approve, what exactly did they approve, and what stops it from running twice?
Summary
An agent that can change things in other systems needs a person in its path for some of those changes, and the design question is which changes and how the checkpoint is built. A rule that looks at what an action does, such as whether it can be undone or how much money it moves, keeps the person’s attention for the actions that matter. An approval is only worth something if it is tied to the exact action the person saw, expires, and is checked by the system that performs the action rather than by the model. The pause has to be stored durably so that a restart does not lose it, and the action that follows has to be safe to retry, which an idempotency key provides. A defensible design states the gate, the binding, the stored state and the retry rule.
Key concepts
| Term | Definition |
|---|---|
| Approval gate | A point in an agent’s path where an action does not run until a person has approved it. |
| Reversible action | An action the system can later undo with another action, for example voiding a refund before it settles. An action that cannot be undone, such as sending an email, is irreversible. |
| Approval rule | The stated test that decides whether an action goes to a person or runs on its own. In this chapter it looks at the action’s effect: its verb, whether it is reversible and its amount. |
| Automation bias | The tendency to rely on a system’s output without checking it. The EU AI Act names it as something people overseeing a high-risk AI system must remain aware of. |
| Fingerprint | A short value computed from the exact parameters of an action, so that a change to a parameter changes the value in practice. Used to tie an approval to what the person saw. |
| Approval lifetime | The period after which an approval no longer authorises anything. This chapter counts it in minutes of a simulated clock. |
| Complete mediation | Checking authorisation in the system that performs the action, on each attempt, rather than relying on the model to decide whether it is allowed. |
| Checkpoint | The agent’s state written to durable storage at the pause, so a process that stops and restarts can resume where it paused. |
| Idempotent | Describes an operation that has the same effect whether it runs once or several times. |
| Idempotency key | A unique value the caller sends with a request so the server can recognise a retry of the same request and return the first result instead of acting again. |
Explanations
The failure, before the explanation
Take a refund agent in front of a support team. It proposes a refund of 250 for order O-9 and shows a person a card: “Approve refund for O-9?” The person approves. Three things then happen on three different days.
On the first day the network drops after the refund system accepted the call but before the agent saw the reply. The agent retries, and the customer is refunded twice. On the second day the agent’s process restarts while a card is waiting, the pending state is held in memory and is gone, and the card is asked again as a new request, so the person has to approve the same refund a second time and cannot tell whether the first one already ran. On the third day the agent re-plans after the approval and calls the refund system with 2,500. The approval record says only “refund for O-9: approved”, the system checks that an approval exists, and it passes.
Nothing in the model was wrong in any of the three. The gate existed. What failed was the checkpoint around it: an approval that named the order but not the amount, a pause that did not survive a restart, and an action that was not safe to repeat. The rest of this chapter builds those three properties, after first deciding where the person belongs.
Where the person sits: gating by effect
Two examples first. A tickets agent adds a label to a ticket. The label can be changed again in a second, and nobody is hurt by a wrong one. The same agent is also able to send an email to a customer. Once sent it cannot be recalled. A second way to see the difference is a lift and a door. A lift can be stopped and sent back down, and a door can be reopened. A sealed envelope that has been posted cannot.
The sorting idea is to gate by what an action does, not by how confident the model sounds. OWASP’s entry on excessive agency, a risk for systems that give a model the ability to act, recommends to “utilise human-in-the-loop control to require a human to approve high-impact actions before they are taken” (OWASP GenAI Security Project, LLM06:2025 Excessive Agency). The entry does not define a threshold, so the team defines one. A workable rule looks at three facts that a system can read without asking the model: the verb of the permission (a read changes nothing), whether the action is reversible, and an amount. This chapter’s rule is: an action whose verb is not one that changes something runs on its own; an irreversible action goes to a person; a reversible action goes to a person when its amount is at or above a limit; anything else runs on its own.
The reason not to send everything to a person is the person. The EU AI Act’s article on human oversight says the people assigned to oversight must be enabled to “remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)” and to “decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output” (Regulation (EU) 2024/1689, Article 14(4), as published at artificialintelligenceact.eu). That text is written for a defined class of systems, high-risk ones, and this chapter does not claim it applies to yours. What it shows is that regulators name over-reliance as a risk, and a queue of hundreds of near-identical cards is a way to produce it.
- Applied use. List the agent’s permissions, mark which change something, then write the rule that splits them: verb, reversibility, amount. Run the rule over a sample of real actions and read the split before trusting it.
- Signal. Wording such as “cannot be undone”, “sent to the customer”, “moves money” and “deletes” points to a gate. “Read-only”, “can be corrected in the next step” and “internal label” point away from one.
- Practical consequence. The rule is code, so its limit and its verbs are reviewable and testable. A gate on everything produces a queue that people clear without reading; a gate on nothing leaves a single wrong call with no checkpoint.
- Versus the distractor. The tempting wrong answer is to let the model rate its own confidence and gate on that. Confidence is not an effect: a confident wrong refund is the case the gate exists for.
What the person approves: the action, the fingerprint and the lifetime
A cheque shows the second way in. A cheque signed with the amount blank can be filled in by anyone who holds it. A cheque with the payee and amount written in, and a date after which the bank refuses it, authorises one payment. A boarding pass works the same way: it is for one named passenger and one flight, and it is checked at the gate by someone other than the traveller.
An approval should be that kind of cheque. When the agent asks for approval it shows the person the exact action: the verb, the target and the amount. The system computes a fingerprint from those parameters and stores it with the approval and the time it was issued. A simple form joins the parameters with a separator and hashes them, written in one fixed order so that the same parameters always give the same text. When the action is later executed, the system recomputes the fingerprint from what it is about to do and compares it with the stored one. If a parameter differs, the fingerprints differ for practical purposes (a hash function can in theory give the same value for two inputs, which is rare enough that this design relies on it), and the execution is refused. That would have stopped the third day’s 2,500 refund.
Two more checks belong with it. The approval names the action it was issued for, so an approval for one action cannot be spent on another. And it has a lifetime: an approval that was true when the card was clicked may not be true when the order has since changed, so after the lifetime the system treats the approval as absent and asks again. A reasonable order of checks when an execution arrives is: if the rule says the action needs no person, allow it; if there is no approval, refuse; if the approval is for another action, refuse; if the parameters differ, refuse; if the approval is older than its lifetime, refuse; otherwise allow. Putting the checks in a fixed order means a refusal always has one stated reason.
Where this check lives matters. OWASP’s mitigation for excessive agency says to “implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed” (OWASP, LLM06:2025). An approval that the agent checks about itself can be skipped by a model that has been misled, for instance by the prompt injection covered earlier in the course. The system that performs the action has no reason to trust the agent’s claim that approval exists, so it looks the approval up.
- Applied use. Store the approval as a record with the action id, the fingerprint of what was shown, the time and the reviewer. Make the executing service, not the agent, read that record.
- Signal. “Approved, then the amount changed”, “approved yesterday” and “the agent said it was approved” point toward binding, lifetime and server-side checking. A stem that gives only “an approval exists” is describing the weak design.
- Practical consequence. The card the person sees and the fingerprint must come from the same parameters. Tying approval to the order alone, as on the third day, lets the parameters drift. Short lifetimes mean more repeat cards, which is a cost the team chooses knowingly.
- Versus the distractor. A tempting wrong answer is to ask the model to confirm that the action it is about to run matches the approval. The check is then made by the party it is meant to constrain.
The pause that survives a restart: the checkpoint
A pause is easier to see as a save point in a game, and as a bookmark with a note of the page. If the console is switched off, the save lets you carry on; a pause held only in the player’s head does not. An agent waiting for a person may wait a long while, and the process that started the wait may not be the one that finishes it.
A checkpoint is the agent’s state saved to durable storage at the moment it pauses. One framework’s documentation describes the mechanism: interrupts “allow you to pause graph execution at specific points and wait for external input before continuing”, they need “a checkpointer to persist the graph state (use a durable checkpointer in production)”, and the thread identifier you choose is “effectively your persistent cursor”, so that reusing it resumes the same checkpoint while a new value starts a new thread with empty state (LangChain, LangGraph interrupts). That is one framework’s design. The general requirement it illustrates is that the waiting state is stored outside the process, keyed so that the answer finds it.
The same page carries a caution that decides the next section. It says interrupts work by re-running the node they were called from, so “side effects called before interrupt should (ideally) be idempotent”, and that code placed before the interrupt will run again on resume, which can create duplicate records. The practical rule is to put the approval request first and the effect after it, so that resuming repeats only harmless work.
- Applied use. Store the pending action and its approval request in a database row or durable state store, with a stable identifier, and resume from that identifier when the person answers.
- Signal. “The service restarted while waiting”, “reviewers answer the next day” and “the pod was replaced” point toward durable state. “The approval page stays open until the person answers” describes state held by a connection.
- Practical consequence. In-memory waiting loses requests silently on a restart. With a checkpoint, the system needs a place to store state and a way to find the waiting request again, which are operations the team now runs.
- Versus the distractor. Holding the request open while a person decides looks simpler and works until a restart or a timeout. A tempting second wrong answer is to reuse a fresh identifier for each resume, which in the framework above starts empty state rather than resuming.
The action that can run twice: idempotency
The failure on the first day was a retry. A lift button shows why a retry is dangerous when the operation is not repeatable: pressing it twice after it has lit does no harm, because the operation is idempotent, while sending two identical orders to a kitchen produces two meals. A numbered order slip turns the second kind into the first: the kitchen sees number 42 again and recognises the same order.
An idempotency key is that slip. The caller generates a unique value for the request and sends it. Stripe’s API documentation describes the design: the server saves “the resulting status code and body of the first request made for any given idempotency key”, and later requests with the same key return the same result; keys can be removed once at least 24 hours old, and a key reused after pruning produces a new request; the server compares incoming parameters with those of the original request and errors if they are not the same (Stripe, Idempotent requests). That is one provider’s behaviour, and another provider’s service may differ, so a design reads the documentation of the service it calls. For an approval flow, a natural key is something that is unique to the approved action, such as the approval’s identifier, because every retry of that action then carries the same key, and a retry that has changed a parameter is caught as a conflict.
A key that is generated fresh on each attempt gives no protection, because the server sees a different key every time. The key has to be created once, stored with the pending action, and reused by each retry.
- Applied use. Create the key when the approval is issued, store it with the pending action, and send it with every attempt to perform the action, including attempts after a restart.
- Signal. “The call timed out but the refund went through”, “the worker restarted and replayed the queue” and “duplicate charge” point toward idempotency. “Fresh identifier on each request” points away from it.
- Practical consequence. The service that performs the action must support keys, or the caller must keep its own record of what was done. Counting real effects against attempts shows whether the design works: three approved actions should produce three effects however many attempts there were.
- Versus the distractor. A tempting wrong answer is to say a person’s approval already makes the action safe. Approval answers whether it may run, and it does not say how many times it has run.
Three situations you will meet, and what they mean for your design
- A support agent proposes to send a customer an email that quotes a refund decision. Nobody can recall the email once it is sent.
- A reviewer leaves a refund card unanswered for a long time, the service is redeployed in the meantime, and the reviewer then approves.
- A payment API times out after the agent calls it, and the agent’s retry logic calls it again.
| What you see | What to do | Why |
|---|---|---|
| Irreversible action, small amount (situation 1) | Send it to a person; the rule gates on reversibility before amount | An email has no amount, and the rule still catches it |
| A long wait across a restart (situation 2) | Store the pending action durably, resume from its identifier, and recheck the approval’s lifetime on execution | The approval may have expired by the time it is used |
| A timeout, then a retry (situation 3) | Send the same idempotency key with the retry | The server can return the first result instead of acting again |
Flashcards
Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.
What does OWASP recommend for high-impact actions taken by an agent?
Human-in-the-loop control, so that a person approves high-impact actions before they are taken. It also says to implement authorisation in downstream systems rather than relying on the model to decide whether an action is allowed. The entry leaves the definition of high-impact to the team.
What three facts does this chapter's approval rule read?
The permission’s verb, whether the action is reversible, and its amount. They can be read by a system without asking the model, so the rule is code that can be reviewed and tested.
Why is gating on the model's own confidence the wrong rule?
Confidence is not an effect. A confident wrong refund is exactly the case a gate exists for, so the rule has to look at what the action does.
What is a fingerprint, and what does it stop?
A value computed from the exact parameters of an action. Stored with the approval and recomputed at execution, it stops an execution whose amount or target differs from what the person saw.
List the checks in order when an execution arrives.
If the rule needs no person, allow. Otherwise refuse when there is no approval, when it is for another action, when the parameters differ, or when it has expired; allow if none of these applies. The fixed order gives each refusal one stated reason.
Why must the executing service check the approval and not the agent?
An agent that has been misled may claim approval exists. The service performing the action has no reason to trust that claim, so it looks the approval up, which is complete mediation.
What is a checkpoint, and what does it need?
The agent’s state saved to durable storage at the pause. It needs a durable store and a stable identifier so that the answer can find the waiting request after a restart. One framework calls its identifier a persistent cursor.
What does the LangGraph documentation warn about code before an interrupt?
It runs again on resume, because the node is re-run, so side effects placed before the interrupt should be idempotent. Put the approval request first and the effect after it.
What does an idempotency key do, and why must it be reused?
It lets the server recognise a retry and return the first result. A fresh key on each attempt looks like a new request every time, so the key has to be created once and reused by every retry.
What does Stripe's page say happens when a key is reused with different parameters?
The idempotency layer compares incoming parameters with those of the original request and returns an error if they differ. That is one provider’s behaviour, and another service may not do the same.
Model answer
The question: “An agent can issue refunds. Where does a human sit in its path, and how do you build that checkpoint so it is trustworthy?” An architect is asked this in a design review before the agent is given the permission. A strong answer hits five beats, in this order:
- Gate — which actions go to a person, by a stated rule on effect.
- Show — what the person sees: the exact verb, target and amount.
- Bind — how the approval is tied to those parameters, checked by the executing system, and expired.
- Persist — where the pending request is stored so a restart does not lose it.
- Once — how a retry is made safe, with a key created once and reused.
“I would gate by effect, not by the model’s confidence: reads and reversible low-amount changes run on their own, and irreversible actions or amounts at or above a limit go to a person, so reviewers see few enough cards to read them. The card shows the exact verb, target and amount. The system computes a fingerprint from those parameters and stores it with the approval, the action’s identifier and the time, and the executing service, not the agent, checks the approval on each attempt: that it exists, is for this action, matches the fingerprint and has not expired. The pending request is stored durably under a stable identifier, so a restart or a long wait does not lose it, and the approval is rechecked when the action runs. The action is sent with an idempotency key created when the approval was issued and reused by every retry, so a timeout followed by a retry produces one effect. I would add that I would read the documentation of the service being called to see how it treats keys.”
Missing a beat is a failure state. Without gate the answer sends everything to a person or nothing. Without show the person approves something they have not seen. Without bind the approval can be spent on something else. Without persist a restart loses the request. Without once a retry repeats the effect.
Why this matters
- Reviews ask for it. A proposal to let an agent change records is judged by what a person sees, what the approval is tied to, and what happens when the process restarts.
- Money and mail do not come back. Refunds, payments and messages are cases where a repeated or altered action cannot be quietly undone, and the design has to make them safe before they run.
- Regulation names the human role. Where a system falls under the EU AI Act’s high-risk rules, the article on human oversight lists abilities the people must have, including overriding the output and stopping the system; whether a system is in scope is a separate question.
- Talking to customers. When a customer asks whether a person approves what the agent does, an architect who can say which actions, what they see and how it is enforced gives a better answer than “yes, there is a human in the loop”.
After this chapter
The project for this chapter is Gate, Bind and Retry (project.md). You will apply an approval rule to nine
pending actions, check eight executions against their approvals, and replay eight attempts against a keyed and
an unkeyed action, finishing by writing executed.tsv.
Chapter 8 opens with the effects recorded in your executed.tsv. It asks where those actions run, which account or
environment they can reach, and how a boundary limits what one compromised environment can do.
Project
Hands-on Project — Gate, Bind and Retry
Objective
Apply an approval rule to an agent’s pending actions, check each execution against the approval it carries, and replay a run of retries against a keyed and an unkeyed action, so that the choice of where a person sits rests on a stated rule and on counted effects.
What this project needs: Python 3 and its standard library only. No task needs a cloud account, a model, an API key or an internet connection. The actions, approvals and attempts are printed below, and every check is an exact comparison. All three tasks are required, and Tasks 1 and 2 are done in order because Task 2 uses the decisions Task 1 produces.
Your evidence file. Keep one plain-text file, chapter7-evidence.txt. Each task tells you which lines to
add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write
what your code printed, not what you expected it to print. Where a value is a list, write its items
comma-separated with no spaces, in the order given.
The data and the rules for all tasks
Copy this into your Python file. A permission is resource:verb. A pending action is
(action_id, agent, permission, target, amount, reversible); reversible is true when the system can later undo
the action with another action.
import hashlib
TRIMMED = { # the previous chapter's trimmed grants
"triage-agent": {"tickets:read", "tickets:write"},
"refund-agent": {"tickets:read", "refunds:create", "email:send"},
"report-agent": {"reports:read", "tickets:read", "users:read"},
}
MUTATING_VERBS = {"write", "create", "delete", "send"}
LIMIT = 100 # a reversible action at or above this amount needs a person
LIFETIME = 30 # an approval is valid for this many minutes after it is issued
ACTIONS = [
("a1", "triage-agent", "tickets:write", "T-101", 0, True),
("a2", "refund-agent", "refunds:create", "O-7", 40, True),
("a3", "refund-agent", "refunds:create", "O-9", 250, True),
("a4", "refund-agent", "email:send", "ana@example.test", 0, False),
("a5", "triage-agent", "tickets:write", "T-102", 0, True),
("a6", "refund-agent", "refunds:create", "O-12", 100, True),
("a7", "refund-agent", "refunds:create", "O-15", 99, True),
("a8", "report-agent", "reports:read", "R-1", 0, True),
("a9", "refund-agent", "refunds:create", "O-20", 500, False),
]
The approval rule. For an action, take the first that applies:
- the permission’s verb (the text after the colon) is not in
MUTATING_VERBS: the decision isauto; reversibleis false: the decision isapprove;amountis at leastLIMIT: the decision isapprove;- otherwise the decision is
auto.
The fingerprint of an action is the first 8 characters of the hexadecimal SHA-256 of the text
permission|target|amount (for example refunds:create|O-9|250), encoded as UTF-8:
hashlib.sha256(text.encode()).hexdigest()[:8].
Task 1 — Apply the approval rule
First, a carry-in. Count the permissions across the three sets in TRIMMED whose verb is in MUTATING_VERBS.
Then write decide(action) following the rule, and print the decision for all nine actions. Write
oversight-policy.tsv: one line per action, in order, with two fields separated by a single tab character: the
action id and its decision.
Add to your evidence file:
t1_carried_mutating: <the number of mutating permissions across the three trimmed grants>
t1_approve_ids: <ids whose decision is approve, in order, comma-separated, no spaces>
t1_auto_count: <number of actions whose decision is auto>
t1_a6: <decision for a6>
Output: one Python file with decide, an oversight-policy.tsv of nine lines, and four evidence lines.
Task 2 — Check each execution against its approval
This task needs the decisions from Task 1. An approval is (approval_id, action_id, permission, target, amount, issued_minute): what the reviewer saw when it was issued. An execution is (exec_id, action_id, permission, target, amount, approval_id, minute): what the system is about to do, with the approval it presents
(None when it presents none).
APPROVALS = {
"p1": ("a3", "refunds:create", "O-9", 250, 0),
"p2": ("a4", "email:send", "ana@example.test", 0, 0),
"p3": ("a6", "refunds:create", "O-12", 100, 0),
"p4": ("a9", "refunds:create", "O-20", 500, 0),
}
EXECUTIONS = [
("e1", "a1", "tickets:write", "T-101", 0, None, 5),
("e2", "a3", "refunds:create", "O-9", 250, "p1", 10),
("e3", "a4", "email:send", "ben@example.test", 0, "p2", 12),
("e4", "a6", "refunds:create", "O-12", 100, "p3", 45),
("e5", "a9", "refunds:create", "O-20", 500, "p3", 20),
("e6", "a9", "refunds:create", "O-20", 500, None, 21),
("e7", "a9", "refunds:create", "O-20", 500, "p4", 30),
("e8", "a3", "refunds:create", "O-9", 2500, "p1", 5),
]
Write check(execution). It returns the first of these that applies, in this order:
- the decision of the execution’s action (from Task 1) is
auto:allowed; - the execution presents no approval:
no_approval; - the approval’s action id differs from the execution’s action id:
wrong_action; - the fingerprint of the execution’s
permission|target|amountdiffers from the fingerprint of the approval’s:changed; - the execution’s
minuteminus the approval’sissued_minuteis greater thanLIFETIME:expired; - otherwise:
allowed.
An approval is not made single-use by this task. Print the result for all eight executions.
Add to your evidence file:
t2_fp_p1: <the fingerprint of approval p1>
t2_results: <the eight results in order e1..e8, comma-separated, no spaces>
t2_allowed_ids: <exec ids whose result is allowed, in order, comma-separated, no spaces>
t2_expired_count: <number of executions whose result is expired>
Output: a check function, and four evidence lines.
Task 3 — Replay the retries
This task needs only the data below. An attempt is (attempt_no, key, amount): one call to a refund service
that carries a key and an amount.
ATTEMPTS = [
(1, "p1", 250), (2, "p1", 250), (3, "p4", 500), (4, "p1", 250),
(5, "p4", 500), (6, "p3", 100), (7, "p4", 500), (8, "p1", 2500),
]
Replay the attempts twice, in order.
- Unkeyed: the service ignores the key, and every attempt produces one effect of its amount.
- Keyed: the service keeps a record of each key it has seen. A key it has not seen produces one effect of the attempt’s amount, and the record stores that amount. A key it has seen, with the same amount, is a replay: no effect. A key it has seen, with a different amount, is a conflict: no effect.
Write executed.tsv: one line per effect in the keyed replay, in the order the effects happened, with three
fields separated by single tabs: the key, the amount of the effect, and the number of attempts in ATTEMPTS
that carried that key (counting replays and conflicts).
Add to your evidence file:
t3_unkeyed_effects: <number of effects in the unkeyed replay>
t3_unkeyed_total: <sum of the amounts of those effects>
t3_keyed_effects: <number of effects in the keyed replay>
t3_keyed_total: <sum of the amounts of those effects>
t3_replays: <number of replays in the keyed replay>
t3_conflicts: <number of conflicts in the keyed replay>
Output: a executed.tsv of one line per keyed effect, and six evidence lines.
What carries forward
Keep executed.tsv, oversight-policy.tsv and chapter7-evidence.txt. The next chapter opens with the effects in
your executed.tsv: the actions that really ran. It asks where they ran, which account or environment they could
reach, and what a boundary around that environment limits.
TechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic or any other company named in this course.