Chapter 5 — Workflow Orchestration for Long-Running Work

Course: Building AI Agents

Prerequisites: You can read and run a short Python script and read a JSON file. You know what an agent loop is (a model proposes an action, your code carries it out, the result goes back; Chapter 3). You know that a tool call is a request your code decides whether to carry out (Chapter 1). Everything else this chapter uses (durable step, checkpoint, retry policy, backoff, jitter, idempotency key, replay) is defined in the notes before it is used.

What the reader can do after this chapter:

• Explain, for a job of several steps that stops part-way, which steps are skipped on resume, which run again, and why a step that already did its work can still run a second time.
• Compute a retry schedule from an initial interval, a backoff multiplier, an attempt limit and a maximum interval, and say what jitter changes about it.
• Design the idempotency key for a side effect (a message, a payment, a ticket) so that a repeated attempt is recognised by the receiver, and say what a stable key is derived from.

Core question this chapter answers: When agentic work runs for longer than one request and something stops it part-way, how does it carry on from where it stopped without losing finished work and without doing the same side effect twice?

Connections:

• Builds on: Chapter 3’s loop, which assumed the whole run stays alive from start to finish. This chapter removes that assumption.
• Leads into: Build and Evaluate RAG Applications, which turns “it seems to work” into a repeatable evaluation suite; the step records and event logs you produce here are the kind of evidence an evaluation reads.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question, so you can check that your answer has the right shape before you submit it.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines. Every line’s accepted value, or the rule that decides it, follows from what the task’s code prints, so you can check your own file before you submit it.

Building AI Agents — Chapter 5: Workflow Orchestration for Long-Running Work

Before you start

Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library. No task needs an account with a model provider, an API key or any payment, and no task calls a model: the work each step does is small fixed code.

Keep one question in mind while you read: if this job stopped right now, between any two lines, what would the next run do, and would anything happen twice? By the end of the chapter you should be able to answer it for any multi-step job from three things: what it records after each step, how it retries, and how the systems it calls recognise a repeated request.

Summary

An agent that researches, drafts, sends a message and files a ticket is not one request; it is a sequence of steps that can take longer than a connection, a process or a deployment stays up. When such a job stops part-way, there are two ways to get it wrong: start over and lose or repeat finished work, or carry on and repeat a side effect that had already happened. Orchestration is the discipline that avoids both. Each finished step has its result recorded in storage that outlives the process, so a restarted run skips what is recorded and continues from the first step that is not. Failed steps are retried on a schedule that spaces attempts out. And because a step can do its work and stop before its record is written, every step that changes something outside your system must be safe to run twice, which in practice means sending the receiver a key that lets it recognise a repeat.

Key concepts

Term	Definition
Workflow	A job made of named steps run in a defined order, where each step’s result can be used by the steps after it.
Step	One unit of work inside a workflow. A step may read data, call a model, or change something in another system (a side effect).
Side effect	A change a step makes outside your own process: a sent message, a created ticket, a charge, a deleted row.
Durable	Stored somewhere that survives the process stopping, such as a file or a database, rather than only in memory.
Checkpoint	The durable record that a step finished, together with its result, written so that a later run can use it instead of repeating the step.
Resumption	Starting a stopped workflow again so that it reads its checkpoints, skips the steps already recorded and continues with the first step that is not.
Retry policy	The rules for trying a failed step again: the delay before the first retry, how fast the delay grows, the largest delay allowed and how many attempts to make.
Backoff	Making each wait before a retry longer than the one before, by multiplying the previous wait by a fixed factor.
Jitter	A random reduction or spread added to each retry wait so that many callers that failed together do not all retry at the same moment.
Idempotent	Safe to repeat: performing the operation several times has the same result as performing it once.
Idempotency key	A unique value a caller sends with a request so that the receiver can recognise a repeat of the same request and return the first result instead of acting again.
Replay	Re-running a workflow’s code against its recorded history so that it reaches the point where it stopped without repeating the recorded steps.

Explanations

The failure, before the explanation

Here is an illustrative job of the kind this chapter is designed for. A team builds an assistant that handles an incoming support ticket in four steps: fetch the ticket, parse its fields, notify the customer with a message, and close the ticket. It runs as one script in one process.

The script gets through fetch and parse. In notify it sends the message to the customer, and then the process is killed: a deployment replaced the server in the same moment. Nothing was written down. The on-call engineer starts the script again.

Two things go wrong, and they are different. The script begins at fetch and does the first two steps again: wasted work here, but for a step that calls a paid model, wasted money and a possibly different answer. Worse, it reaches notify and sends the message a second time. The customer now has two copies of the same message, and the ticket system has no way to say which one the assistant “meant”. Neither fault is in a model. Both come from a job with several steps and no record of how far it got, calling a system that cannot tell a repeat from a new request.

Durable steps: write down what is finished

Start with the fix for the first fault, in the same example. After each step finishes, the script writes one line to a file: the step’s name and its result. When it starts, it reads the file first. If fetch and parse are in the file, it takes their results from there and goes straight to notify. That file is a checkpoint store, and the run after the crash uses it to skip finished work.

A second way to picture it: a build pipeline that caches the output of each stage. You do not recompile a module whose cached output is still valid; you reuse the output and run only the stages that have no cached result. A checkpoint is the same idea, with one difference: the pipeline cache is an optimisation, while a workflow’s checkpoints are the only record of what already happened, so they must live in storage that outlives the process.

Production systems provide this for you. Temporal’s documentation describes its event history as “a complete and durable log of everything that has happened in the lifecycle of a Workflow Execution” (Temporal, Event History). A hand-written file, as in this chapter’s project, is the smallest version of the same idea.

Two details decide whether a checkpoint store works:

• Write the record after the step finishes, and write it atomically. A record written before the step runs claims work that has not happened. A record half-written when the process dies corrupts the file. One way to avoid the second problem is to write the new content to a temporary file in the same folder and then rename it over the old one. Python’s documentation says a successful rename “will be an atomic operation (this is a POSIX requirement)” (Python, os — Files and Directories). That protects the file name from holding a half-written copy; it is not a promise that the data has reached the disk after a power loss.
• Record the result, not only the fact. Later steps need parse’s output. If the checkpoint holds only “parse finished”, the resumed run must repeat parse anyway.

Resumption and replay: carrying on from where it stopped

Take the four-step ticket job again, now with checkpoints. The process is killed during notify. The checkpoint file holds fetch and parse. On the next run the workflow reads the file, skips those two and runs notify: the first step with no record.

That is the whole mechanism in its simplest form, and notice what it does not say. It says that a step with a record is skipped. It says nothing about a step that started and did not finish, because that step has no record, and the run cannot tell from the file whether its work happened. That gap is what the rest of the chapter is about.

Workflow engines add a second mechanism on top: replay. Temporal describes it like this: “A Replay is the method by which a Workflow Execution resumes making progress. During a Replay the Commands that are generated are checked against an existing Event History”, and “If a failure occurs, the Workflow Execution picks up where the last recorded event occurred in the Event History” (Temporal, Workflow Execution). In plain terms: the workflow’s own code runs again from its start, and every time it reaches a step the history already holds, it takes the recorded result instead of doing the work.

A second way to see replay: it is like re-reading a game’s move log to rebuild the board. You do not play the moves again against an opponent; you apply each recorded move until you reach the last one, then play on. For that to work the rebuild must come out the same every time, which is why the same documentation says “Workflow code must be deterministic to support replay, and failure-prone or non-deterministic operations (API calls, LLM invocations, etc.) should be placed in Activities, which have built-in retry support” (Temporal, Retry Policies). A model call returns different text on different runs (as taught in Getting Started with AI), so it belongs in the retried, recorded part of such a system, not in the code that replay re-runs.

Retries: spacing out the second attempt

Here is a step that calls a service that is briefly overloaded. The first attempt fails. If your code retries immediately, in a tight loop, a hundred workflows that failed together send a hundred more requests at the same instant, which is what overloaded the service in the first place.

A retry policy spaces the attempts. One provider’s documentation defines the knobs for its workflow service (AWS, Error handling in Step Functions): IntervalSeconds, the wait before the first retry (1 by default); MaxAttempts, the number of retries (3 by default, and “A value of 0 specifies that the error is never retried”); and BackoffRate, “The multiplier by which the retry interval denoted by IntervalSeconds increases after each retry attempt” (2.0 by default). Its own worked example uses an interval of 3, a MaxAttempts of 3 (three retries) and a rate of 2: the first retry takes place three seconds after the error, the second six seconds after the first retry, and the third 12 seconds after the second.

That is a schedule you can compute without running anything: start with the interval and multiply by the rate for each further retry, so the waits are 3, 6 and 12.

Backoff grows quickly, so a policy also needs a ceiling. The same page offers MaxDelaySeconds, which “limits the exponential wait times resulting from the backoff rate multiplier”; if you do not set it, Step Functions does not limit the wait. Temporal’s retry policy has the same two ideas under other names: a backoff coefficient (default 2.0) and a maximum interval, which its documentation describes as preventing “the retry interval from growing infinitely” (Temporal, Retry Policies). With an initial wait of 1, a rate of 2, six retries and a ceiling of 10, the waits are 1, 2, 4, 8, 10 and 10: the fifth wait would have been 16 and is cut to the ceiling. Count carefully when you compare systems: Step Functions’ MaxAttempts counts retries, while Temporal’s maximum attempts counts executions, and its documentation says “Setting the value to 1 means a single execution attempt and no retries” (Temporal, Retry Policies). Six Temporal attempts would therefore give five retry waits.

Jitter handles the other half of the overload example. Step Functions defines it as spreading retries “over a randomized delay interval”. With FULL jitter, each wait is randomised between 0 and the interval it would otherwise have been, so a first retry that would have waited 2 becomes a value between 0 and 2. The default is NONE. A hundred callers that failed together now retry at a hundred different moments instead of one.

Defaults differ. Temporal retries a failed Activity by default and, among its defaults, sets maximum attempts to unlimited; Step Functions’ default is three retries. A policy you copy from one system into another can behave very differently, so state each of the four numbers instead of relying on a default.

Retrying is only safe for a step that is safe to repeat. That is the next section.

Idempotency: a repeated step must not repeat its effect

Return to the notify step. It sent a message, the process died, the run was restarted and the step ran again. A retry schedule would cause the same thing: the first attempt may have succeeded and the failure may have been in reading the reply, so retrying sends a second message.

The property that fixes this is idempotency. Temporal’s documentation says “Temporal recommends that Activities be idempotent. Idempotence means that performing an operation multiple times has the same result as performing it once” (Temporal, Activity Definition), and gives infrastructure provisioning as an example where a repeated call would lead to unwanted cost.

Some operations are idempotent by nature: setting a ticket’s status to closed has the same result however many times you do it. Others are not: appending a message to a customer’s inbox adds one each time. For those, the caller sends an idempotency key. Stripe’s documentation describes it: “A client generates an idempotency key, which is a unique key that the server uses to recognize subsequent retries of the same request.” Its server saves “the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails”, and “Subsequent requests with the same key return the same result, including 500 errors” (Stripe, API reference: Idempotent requests). The effect happens once; every repeat gets the first answer.

A second way to see it: a numbered cloakroom ticket. You hand over a coat and get ticket 17. If you lose track of whether the attendant took the coat and hand it over again with the same ticket, the attendant looks up 17, sees there is already a coat and does not hang a second one. The ticket number is the key, and the attendant, not you, is the one who keeps the record.

Three rules follow from Stripe’s description, and each is a failure you can build by getting it wrong:

• The key must be the same on every attempt of the same operation. Stripe says how you create keys is up to you and suggests V4 UUIDs, but a key that is generated again on every attempt is a new key each time, so the receiver sees four different requests. Generate it once per operation and store it with the checkpoint, or derive it from values that do not change, such as the workflow’s run id and the step name (run-001:notify).
• The same key with different content is an error, not a repeat. Stripe’s idempotency layer “compares incoming parameters to those of the original request and errors if they’re not the same to prevent accidental misuse.” A key identifies one specific request.
• A saved result exists only once the work began. Stripe saves results “only after the execution of an endpoint begins”; a request that failed validation, or that conflicted with one still executing, is not saved, and may be retried (Stripe, API reference: Idempotent requests).

The receiver keeps the record. If the receiver is a system you do not control and it has no key feature, you cannot make the step idempotent by sending a key. What you can do is keep your own record of the attempt, with a stable message id, and check it before repeating. Here is the trade-off in that design, stated as reasoning rather than as vendor documentation: if you write “about to send” before sending, a crash in between leaves a record of a send that never happened, and the step may never run again (a possible miss). If you send first and write the record after, a crash in between leaves a send with no record, and the step repeats it (a possible duplicate). You cannot close both gaps without help from the receiver, so decide which is worse for that message: a missing reminder email and a duplicate refund are not the same problem.

Putting it together: what the ticket job should look like

The ticket job now has all three layers, and each answers a different failure.

1. Checkpoints answer “where did it stop?”: fetch and parse are recorded, so the next run begins at notify.
2. A retry policy answers “what if the call fails for a moment?”: spaced attempts with a ceiling, and jitter if many workflows share a dependency.
3. An idempotency key answers “what if the step ran but its record was not written?”: notify sends run-001:notify with the message, so the second run’s send is recognised and not delivered again.

Remove any one and the matching failure comes back. The key without checkpoints still repeats cheap work and paid model calls. Checkpoints without the key still send two messages in the case the example began with.

Three situations you will meet, and what they mean for your design

Before the table, three situations:

1. A QA engineer’s batch of two hundred summarisation steps dies at step 140 because a key expired. The team reruns it and pays for steps 1–139 again.
2. A backend engineer adds retries to a “create refund” call. During a network blip the first call succeeds but its reply is lost; the retry creates a second refund.
3. A DevOps engineer gives each retry of a “post to channel” step a fresh random id, to make sure “each message is unique”. The channel receives one copy per attempt.

What you see	What the design allowed	What to do instead
A rerun repeats finished, paid work (situation 1)	No durable record of finished steps	Checkpoint each step’s result after it finishes, and skip recorded steps on resume
A retry repeats a side effect (situation 2)	A non-idempotent call was retried	Send an idempotency key with the request, or use an operation that is idempotent by nature
One copy per attempt (situation 3)	The key changed on every attempt, so the receiver saw new requests	Generate the key once per operation, or derive it from the run id and step name

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

What does a workflow record after a step finishes, and why the result as well as the fact?

A checkpoint: the step’s name and its result, in storage that outlives the process. Later steps need the result as input; if only the fact is stored, the resumed run has to repeat the step to get it.

A workflow is restarted after a crash. Which steps does it skip?

The steps that have a checkpoint. It runs the first step without one. A step that started but has no checkpoint is treated as not finished, even if its work happened.

Why can a step run twice even when checkpoints work?

The step can do its work and stop before its checkpoint is written. The record then says “not finished”, so the resumed run does the work again. That is why steps with side effects must be safe to repeat.

What is replay in a workflow engine?

Re-running the workflow’s code against its recorded history, taking recorded results instead of repeating the steps, until it reaches the point where it stopped. The code that replay re-runs must come out the same each time.

Why does an LLM call belong in a retried step and not in replayed workflow code?

A model’s output can differ between runs, and replay needs the code to behave identically. One provider’s documentation says failure-prone or non-deterministic operations, naming LLM invocations, belong in Activities, which have built-in retry support.

Compute the waits for an initial wait of 3, a rate of 2 and three retries.

3, 6 and 12: each wait is the previous one multiplied by the rate. This is the provider’s own worked example.

What does a maximum interval do to a backoff schedule?

It caps each wait. With an initial wait of 1, a rate of 2 and a cap of 10, the waits are 1, 2, 4, 8, 10, 10: the fifth would have been 16 and is cut to 10, and so is every later one.

What problem does jitter solve, and what does full jitter do to a wait of 8?

Many callers that failed together retry together. Full jitter replaces each wait with a random value between 0 and what the wait would have been, so a wait of 8 becomes a value from 0 to 8 and the callers spread out.

What is an idempotency key, and who keeps the record of it?

A unique value sent with a request so the receiver can recognise a repeat. The receiver keeps the record: it saves the first result for the key and returns it for later requests with the same key.

Your retry generates a fresh random key for every attempt. What happens?

The receiver sees a new key each time and treats every attempt as a new request, so the side effect happens once per attempt. The key must be the same across all attempts of one operation.

Model answer

The question: “Our assistant runs a six-step job that sends a customer email and files a ticket. The server restarted in the middle of it. Walk me through how it should carry on, and how you make sure the customer gets one email.” This is the question a lead engineer asks before a long-running agent goes live. A strong answer hits five beats, in this order:

1. Checkpoint — each finished step has its result recorded in durable storage.
2. Resume — the restarted run skips recorded steps and starts at the first one without a record.
3. Gap — a step can do its work and stop before its record is written, so it may run twice.
4. Retry — failed steps are retried on a stated schedule: interval, multiplier, ceiling, attempt limit, and jitter where many callers share a dependency.
5. Key — the email step sends an idempotency key that is the same on every attempt, so the receiver recognises a repeat.

“I’d record each step’s result in durable storage as soon as the step finishes, so the job can be restarted without losing finished work. On restart it reads those records, skips the steps it finds and starts at the first step with none. But there’s a gap: the email step can send the email and be killed before its record is written, so on restart it looks unfinished and runs again. Retries have the same effect. So failed steps get a retry schedule I can state in four numbers, the first wait, the multiplier, a ceiling and a maximum number of attempts, with jitter if many jobs call the same service. And the email step sends an idempotency key made from the run id and step name, the same on every attempt, so the mail service recognises the second send as a repeat and doesn’t deliver it. If the mail service couldn’t take a key, I’d keep my own record of the send and decide whether a possible duplicate or a possible miss is worse for that message.”

Missing a beat is a failure state. Without checkpoint the answer has no way to avoid starting over. Without resume the records are never used. Without gap the answer promises “exactly once” that the design cannot deliver. Without retry a short failure kills the whole job. Without key the second attempt sends a second email.

Why this matters

• Cost. A job restarted from its first step pays again for every model call it already made.
• Customer trust. A repeated message, charge or ticket is visible to people outside your team, and cannot be taken back by rerunning the job.
• Reviews. A reviewer who asks “what happens if this crashes at step four?” is asking for the checkpoint, the gap and the key. An engineer who can answer from the design is better placed to get a long-running agent approved.
• Talking to customers. When a customer asks “can it send my invoice twice?”, the answer is in your keys and your records, and you can show it.

After this chapter

The project for this chapter is Make a Job Survive a Crash (project.md). You will compute retry schedules, build a small workflow runner that records each step’s result in a JSON file and resumes after a simulated crash, and then show a repeated side effect being delivered twice and once.

Build and Evaluate RAG Applications starts from the records this chapter produces. It turns “the job seems to work” into an evaluation suite that fails loudly when a change breaks something, and a step record is exactly the kind of evidence a suite reads.

Hands-on Project — Make a Job Survive a Crash

Objective

Compute retry schedules, build a small workflow runner that records each step’s result in a JSON file and carries on after a simulated crash, and then show a repeated side effect being delivered twice and once, so that you can point at exactly what protects a long-running job from losing finished work and from repeating itself.

What this project needs: Python 3 and nothing beyond its standard library (json, os, random, uuid, collections). No task needs an account with a model provider, an API key or any payment, and no task calls a model or waits on a clock: the “work” each step does is fixed code given below, and retry waits are computed, never slept. All three tasks are required.

Your evidence file. Keep one plain-text file, chapter5-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print.

Task 1 — Compute retry schedules

Write backoff(initial, rate, retries, max_wait=None). It returns a list of retries waits. The first wait is initial; each later wait is the previous uncapped wait (the value before any cap) multiplied by rate; and when max_wait is given, any wait above it is replaced by max_wait. These are the same ideas as the interval, backoff rate and maximum delay fields of a workflow service’s retry settings (AWS, Handling errors in Step Functions workflows), whose own worked example is an interval of 3, three retries and a rate of 2.

Write full_jitter(waits, rng). It returns a list in which each wait w is replaced by rng.random() * w, in order, drawing one number per wait. This is the “between 0 and the interval” behaviour that the same page describes for full jitter.

Call them as follows and print the results:

1. backoff(3, 2, 3)
2. backoff(1, 2, 6, 10)
3. backoff(2, 1.5, 4)
4. full_jitter(backoff(3, 2, 3), random.Random(2026))

Fixing the seed makes the run repeatable, so it can be marked exactly. The Python documentation says that “by reusing a seed value, the same sequence should be reproducible from run to run as long as multiple threads are not running”, and that random() “will continue to produce the same sequence” when the same seed is given to a compatible seeder, even though other functions in the module may change between versions (Python, random — Notes on Reproducibility). So use only rng.random(), in one thread.

Add to your evidence file. Write every number with exactly two decimal places, comma-separated, no spaces. Numbers are compared to the key within 0.01:

t1_a: <the waits from call 1>
t1_b: <the waits from call 2>
t1_c: <the waits from call 3>
t1_total_a: <the sum of the waits from call 1>
t1_total_b: <the sum of the waits from call 2>
t1_total_c: <the sum of the waits from call 3>
t1_jitter: <the three values from call 4>
t1_jitter_within_bounds: <true if every jittered value is between 0 and its original wait, else false>

Output: a script with two functions and eight evidence lines.

Task 2 — Build a runner that resumes

Write run_workflow(steps, path, calls). steps is a list of (name, function) pairs, path is the checkpoint file and calls is a collections.Counter. The runner does this:

1. If path exists, load it as JSON; otherwise start with {"done": {}}. The done dictionary maps each finished step’s name to its result.
2. For each step in order: if its name is already in done, add its name to a list called skipped and move on. Otherwise add 1 to calls[name], call function(done) (the function receives the dictionary of results so far), store the result in done[name], and write the whole state to path before going to the next step. Write to a temporary file such as path + ".tmp" and then call os.replace to put it in place, so that a stop in the middle of writing cannot leave a half-written checkpoint under the real name. The Python documentation says a successful rename “will be an atomic operation (this is a POSIX requirement)” and that a rename may fail when the two paths are on different filesystems (Python, os — Files and Directories), so keep the temporary file in the same folder. os.replace is the same rename that also overwrites an existing file. This protects the file’s name from holding a half-written copy; it does not promise that the data has reached the disk if the machine loses power, which is outside this project.
3. Return skipped.

Use these four steps. outbox stands for a customer’s inbox, outside your workflow. crash_now is a dictionary you set before each run.

class Crash(Exception):
    pass

outbox = []
crash_now = {"on": True}

def fetch(done):  return "T-42"
def parse(done):  return done["fetch"].lower()
def notify(done):
    outbox.append("notify:" + done["parse"])   # the side effect happens here
    if crash_now["on"]:
        raise Crash()                          # the process "dies" after the side effect
    return "sent"
def close(done):  return "closed:" + done["notify"]

STEPS = [("fetch", fetch), ("parse", parse), ("notify", notify), ("close", close)]

Run it twice against the same checkpoint file, starting with no file and an empty outbox and a fresh Counter. First run with crash_now["on"] = True and catch Crash; then load the file and note which step names it holds. Then run again with crash_now["on"] = False.

Add to your evidence file. Step names are written in the order the file holds them, comma-separated, no spaces:

t2_run1_outcome: <crashed or completed>
t2_state_after_run1: <the step names in the checkpoint file after the first run>
t2_run2_outcome: <crashed or completed>
t2_run2_skipped: <the list the second run returned>
t2_calls: <fetch=N,parse=N,notify=N,close=N with the counts from the Counter, in that order>
t2_outbox_len: <the number of items in outbox at the end>
t2_final_result: <the stored result of the close step>

Look at the last three lines before you go on: the second run did not repeat fetch or parse, and yet something was repeated. Do not fix it in this task.

Output: a runner with a checkpoint file, and seven evidence lines.

Task 3 — Stop the repeat with a key

Add a receiver that lives outside your workflow, as a mail service would. Write a class Receiver with a dictionary of seen keys and two counters, calls and deliveries. Its method receive(key, message) adds 1 to calls, then:

• if key has not been seen: add 1 to deliveries, remember (message, "delivered") for the key and return "delivered";
• if key has been seen with the same message: return the remembered result, "delivered", and deliver nothing;
• if key has been seen with a different message: raise ValueError.

This is the behaviour Stripe describes for its idempotency keys: it saves the first result for a key and returns it for later requests with the same key, and it errors when the parameters differ (Stripe, API reference: Idempotent requests).

This task stands on its own: it needs nothing from Task 2. Write send_with_retries(receiver, make_key, message, attempts), which calls receiver.receive(make_key(n), message) for n = 1 up to attempts and returns the list of results. It stands for a step whose first reply was lost, so the step is tried again.

Run these three scenarios:

1. Stable key. Create one Receiver. Call send_with_retries(receiver, lambda n: "run-001:notify", "notify:t-42", 2). Keep the returned list.
2. Random key. Create a new Receiver. Call send_with_retries(receiver, lambda n: str(uuid.uuid4()), "notify:t-42", 2), so every attempt carries a new key.
3. Mismatch. On the receiver from scenario 1, call receive("run-001:notify", "notify:other") and catch the exception.

Add to your evidence file:

t3_stable_results: <the list returned in scenario 1, comma-separated, no spaces>
t3_stable_deliveries: <deliveries on the scenario 1 receiver after scenario 1>
t3_stable_calls: <calls on the scenario 1 receiver after scenario 1>
t3_random_deliveries: <deliveries on the scenario 2 receiver>
t3_random_calls: <calls on the scenario 2 receiver>
t3_mismatch: <raised if the call raised ValueError, else accepted>
t3_extra_deliveries: <random deliveries minus stable deliveries>

Output: a Receiver class, send_with_retries, three scenarios, and seven evidence lines.

What carries forward

Keep your runner from Task 2, your Receiver class and chapter5-evidence.txt. The next chapter opens with the checkpoint file your runner produced and the delivery counts from Task 3: it treats each recorded step result as evidence a test can read, and asks how you would check, automatically and on every change, that a resumed run ends in the same state as an uninterrupted one.