Retries and timeouts in agent loops

Day 24: Retries and Timeouts in Agent Loops: A Retry Is Not a Replay

Your agent can refund the same customer twice and reason well both times. The cause isn't the model. It's the `max_retries=3` and 30-second timeout you copied from a web client. When a tool changes something (a refund, an email, a deploy), those two settings decide whether it runs once, runs twice, or runs twice with different arguments. Most agent bugs I've debugged in production weren't bad reasoning. They were good reasoning repeated at the wrong moment.

The hook: six log lines, two refunds

Here's a trimmed trace from a support agent with a refund tool. The model call has `max_retries=3`. The tool call has a 30-second timeout. Read it top to bottom.

10:02:14.120  agent     step=3  call refund(order='A-1042', amount=49.00, reason='damaged')          idem=7f3a…
10:02:44.121  agent     step=3  TIMEOUT after 30s, error appended to messages
10:02:46.870  agent     step=3  call refund(order='A-1042', amount=54.95, reason='damaged + shipping') idem=c91e…
10:02:47.410  payments  refund ok  id=re_8812  amount=54.95
10:02:55.302  payments  refund ok  id=re_8807  amount=49.00   # the first request, committed at t+41s
10:02:55.900  agent     step=3  final answer: 'Refund issued.'

Which line is the bug? None of them, taken alone. The timeout fired on schedule. The model read the error and made a sensible call: add shipping and try again. The payments API saw two different idempotency keys and did two different refunds. The customer got $103.95 back on a $49 order. The bug is in the design. Line 3 looks like a retry but it's a new action, and nothing in the system could tell the two apart.

The rule for today: a retry is not a replay

In classic distributed systems, a retry means sending the same bytes again. In an agent loop, "try again" means asking a stochastic planner to decide again, and this time it knows more, because its context now includes the failure.

| HTTP / SDK retry                              | Agent retry                                          |
|-----------------------------------------------|------------------------------------------------------|
| Same request, same payload, same key          | New prompt: original context + the error message     |
| Deterministic: attempt 2 == attempt 1         | Sampled: attempt 2 can change args, tool or plan     |
| Safe if the endpoint is idempotent            | Safe only if you make the *decision* idempotent      |
| Bounded by a retry policy                     | Bounded by whatever the loop's max_steps happens to be |
| Lives below the business logic                | IS the business logic                                |

Why the second attempt is a different action

Go back to the tool-calling loop from Day 18. The model proposes a call, you run it, you append the result, you call the model again. When the tool fails, the "result" you append is the error, so the next model call starts from a different prompt. Three things stack up:

• ▹The context changed. "Timeout after 30s" or a stack trace is new evidence, and the model is trained to adjust its plan when it sees new evidence.
• ▹Sampling isn't deterministic. Providers don't promise identical outputs even at temperature 0, and at 0.7 you're asking for variation.
• ▹Models want to help. A failed attempt reads as a hint that the last arguments were wrong, so the model "fixes" them. In the trace it added shipping. On another run it might switch to `issue_store_credit` or split the refund into two calls.

So the model isn't repeating step 3. It's taking a new step 3 that looks a lot like the old one. If that step has side effects, you now have two independent actions in flight.

The timeout that succeeds: orphaned work

A timeout is the caller deciding to stop waiting. The work keeps going. In the trace, the orchestrator gave up at 30s and the payment processor committed at 41s anyway. That refund is orphaned work: a side effect nobody is waiting for, which the agent will never hear about unless it goes and asks.

Agent stacks make this worse. Tool calls are slow and pass through several hops (agent → orchestrator → MCP server → third-party API). Each hop has its own timeout, often set by a different person. The shortest timeout in the chain decides when the agent gives up. The slowest hop decides when the work actually lands.

A quick aside on why you can't fully fix this. Over an unreliable channel, a caller and a worker can never be sure they agree on whether something happened. That's the two generals problem, one of the oldest results in distributed systems and multi-agent coordination. Any acknowledgement can be lost too, so adding more messages never gets you to certainty. Don't chase certainty. Make the action safe to repeat (idempotency) and make the outcome something you can look up afterwards (a lookup handle). Every pattern below builds on those two ideas.

Pattern 1: Split retries into two layers

Most agent code mixes two kinds of retry together. Separate them:

• ▹Layer A is deterministic and sits below the model, inside the tool wrapper (and inside the SDK for the model call itself). It freezes the payload once and derives the idempotency key from (task_id, step_id), never from the attempt number. It retries transient errors only, with exponential backoff and jitter. Every attempt sends the same bytes with the same key.
• ▹Layer B is where the model makes decisions. The model sees a failure only after Layer A has given up, and it gets a structured result instead of a raw stack trace, something like `{status: 'UNKNOWN', ref: 'task-81:refund:A-1042', hint: 'confirm before retrying'}`. Whatever the model decides next is a new action. It isn't a retry.

If you remember one detail from this page, make it step_id. It has to name the business intent, like `refund:A-1042`. It can't name the attempt (`attempt-2`), and it can't be the LLM's tool_call_id. Tool call IDs change on every model turn, so a key built from one changes the moment the model re-plans. With an intent-based key, both calls in the hook trace share a key. A payments API like Stripe will then either return the original result or reject the second request because its parameters don't match. Both outcomes give you one refund.

import random, time, hashlib

TRANSIENT = (ConnectionError, RateLimited, ServiceUnavailable)

def idem_key(task_id: str, step_id: str) -> str:
    # Stable across attempts AND across agent re-plans of the same intent.
    return hashlib.sha256(f'{task_id}:{step_id}'.encode()).hexdigest()[:32]

def call_with_retry(fn, payload: dict, *, key: str, budget, attempts: int = 3):
    frozen = dict(payload)                      # built once, never re-derived
    for attempt in range(attempts):
        budget.check()                          # raises if deadline or spend is gone
        try:
            return fn(**frozen, idempotency_key=key, timeout=budget.slice(0.5))
        except TRANSIENT:
            if attempt == attempts - 1:
                raise
            time.sleep(min(8.0, 0.5 * 2 ** attempt) * random.uniform(0.5, 1.5))

Pattern 2: A timeout means cancel and confirm

When a deadline fires, don't tell the model "failed". You don't know that. Return PENDING with a handle for looking up the outcome later. Before the agent is allowed to act on that step again, it has to confirm what actually happened: query by idempotency key, check the ledger, or reconcile with the downstream system. Cancel the downstream work as well if the API lets you, but never assume the cancel worked.

import asyncio
from dataclasses import dataclass, asdict

@dataclass
class ToolResult:
    status: str                 # 'OK' | 'PENDING' | 'FAILED' | 'BLOCKED'
    data: dict | None = None
    handle: str | None = None   # what to look up to learn the real outcome
    message: str = ''
    def to_json(self): return asdict(self)

async def with_deadline(coro, *, seconds: float, handle: str, ledger) -> ToolResult:
    task = asyncio.ensure_future(coro)
    # Record the late outcome when it lands, so orphaned work stops being invisible.
    task.add_done_callback(lambda t: ledger.record_late(handle, t))
    try:
        return await asyncio.wait_for(asyncio.shield(task), timeout=seconds)
    except asyncio.TimeoutError:
        # We stop WAITING. The work may still commit downstream.
        return ToolResult(
            status='PENDING', handle=handle,
            message=f'No answer in {seconds:.0f}s. Outcome unknown. '
                    f'Call confirm("{handle}") before taking this action again.')

def confirm(handle: str) -> ToolResult:
    r = payments.find_by_idempotency_key(handle)
    if r is None:
        return ToolResult('FAILED', handle=handle, message='Not found; safe to retry with the SAME key.')
    return ToolResult('OK' if r.succeeded else 'PENDING', data=r.summary(), handle=handle)

One nuance. If the downstream honours idempotency keys, a Layer A retry with the same key doubles as a safe confirmation, because the server dedupes it. If it doesn't (plenty of internal APIs, SMTP, most webhooks), you need an explicit lookup or your own ledger. That's why the classification table below says "confirm, never blind retry" for unknown outcomes.

Pattern 3: Budgets, not counts

Retry counts multiply across layers. Say the agent loop allows 3 re-plans, the orchestrator retries the tool step 3 times, the tool wrapper retries 3 times and the HTTP SDK retries 3 times. One bad minute at a dependency turns into 3 × 3 × 3 × 3 = 81 downstream calls, and some of them may be model calls that burn tokens. Nobody picked 81. It's what you get from four defaults set by four different people.

Instead, give the whole task one deadline and one spend budget. Create them at the top and pass them down, the way gRPC propagates deadlines through a call chain. Each layer takes a slice of what's left and never starts its own fresh 30 seconds. Once the budget is gone, every layer stops, whatever its local retry count says. Day 26 covers cost control in depth. Today, just make sure a budget object exists and gets passed down.

from dataclasses import dataclass
import time

@dataclass
class Budget:
    deadline: float          # absolute, time.monotonic()
    usd_left: float
    def remaining(self) -> float: return self.deadline - time.monotonic()
    def ok(self) -> bool: return self.remaining() > 0 and self.usd_left > 0
    def slice(self, frac: float) -> float: return max(0.0, self.remaining() * frac)
    def check(self):
        if not self.ok(): raise BudgetExhausted()
    def charge(self, usd: float): self.usd_left -= usd

Pattern 4: Classify before you retry

Every failure goes into exactly one of four buckets, and each bucket has a single owner:

| Class                       | Examples                                         | Who handles it        | Action                                                        |
|-----------------------------|--------------------------------------------------|-----------------------|---------------------------------------------------------------|
| Transient                   | 429, 503, connection reset, DNS blip            | Layer A               | Retry same payload + same key, backoff + jitter, within budget |
| Ambiguous / unknown outcome | Timeout, 502 after send, dropped connection mid-write | Layer A then confirm | Return PENDING + handle; confirm by key; NEVER blind retry     |
| Model error                 | Bad args, schema violation, wrong tool, 4xx validation | Layer B (the model) | Let it re-plan; a side-effecting re-plan is a NEW action with its own approval gate |
| Permanent                   | 403, account closed, business rule refused, budget exhausted | Human           | Stop, escalate with the trace and the handle                  |

The usual mistake is filing ambiguous outcomes under transient. A timeout on a read is transient. A timeout on a write is ambiguous. Have side-effecting tools declare it (`side_effecting=True`) so the wrapper knows which rule to apply.

Before and after: the refund loop

Before. This is the loop that produced the hook trace, and it's close to what most tutorials show:

def run_agent(task):
    messages = [SYSTEM, user_msg(task)]
    for _ in range(10):                                         # count, not budget
        resp = llm.chat(messages, tools=TOOLS, temperature=0.7,
                        max_retries=3, timeout=30)
        if not resp.tool_calls:
            return resp.text
        for call in resp.tool_calls:
            try:
                result = TOOLS_IMPL[call.name](**call.args, timeout=30)
            except Exception:
                result = 'Error: ' + traceback.format_exc()     # raw trace -> re-plan
            messages.append(tool_result(call.id, result))
    return 'gave up'

# inside the refund tool:
#   payments.refund(order, amount, idempotency_key=str(uuid.uuid4()), max_retries=3)

After. Same loop, with the four changes marked ① to ④:

async def run_agent(task, budget: Budget):                        # ④ one budget for the task
    messages = [SYSTEM, user_msg(task)]
    while budget.ok():
        resp = llm.chat(messages, tools=TOOLS, timeout=budget.slice(0.2))  # SDK retries = Layer A
        if not resp.tool_calls:
            return resp.text
        for call in resp.tool_calls:
            tool = TOOLS_IMPL[call.name]
            key = idem_key(task.id, tool.step_id(call.args))     # ① intent-based, not attempt-based
            prior = ledger.get(key)
            if prior and prior.status in ('OK', 'PENDING'):      # ② confirm before acting again
                result = tool.confirm(key)
            elif tool.side_effecting and ledger.has_attempt(task.id, tool.name) \
                    and not approvals.allow(task, call):         # ④ re-plan = new action, gated
                result = ToolResult('BLOCKED', message='New side-effecting attempt needs approval.')
            else:
                ledger.start(key, call.args)
                result = await with_deadline(                    # ② timeout -> PENDING + handle
                    tool.execute(call.args, key=key, budget=budget),  # ③ Layer A retries inside
                    seconds=budget.slice(0.5), handle=key, ledger=ledger)
                ledger.finish(key, result)
            messages.append(tool_result(call.id, result.to_json()))   # ③ structured, no stack trace
    return escalate(task, reason='budget exhausted', ledger=ledger)  # permanent -> human

• ▹① A stable idempotency key. For a refund, `step_id` is `refund:{order_id}`, so when the model re-plans with amount=54.95 it lands on the same key, and the ledger (or the payments API) catches the duplicate.
• ▹② Cancel and confirm. A timeout returns PENDING plus a handle. Next time the model reaches for that step, the loop confirms the real outcome first instead of executing again.
• ▹③ Layer A stays hidden and failures come back structured. Transient retries happen inside `tool.execute` with a frozen payload. The model only ever sees `{status, handle, message}`, never a traceback, so there's nothing for it to over-correct.
• ▹④ A budget and a gate. One deadline and one spend limit govern every layer. A second side-effecting action from a re-plan needs policy or human approval, and an exhausted budget escalates rather than looping.

Now replay the hook trace against this version. At 10:02:44 the model sees PENDING with a handle. If it re-plans to 54.95, the call maps to the same key, the loop confirms, and at 10:02:55 the ledger records re_8807 as OK. One refund. The model then tells the customer what actually happened: $49 was refunded, and shipping needs a separate decision.

Today's exercise: audit your own agent

Take the agent you built on Days 15–23 and write down answers to these five questions. Any "I don't know" is your next bug.

• ▹Where is every timeout set, and who chose it? Cover the model SDK, orchestrator, tool wrapper, HTTP client and any MCP server. Note the shortest timeout and the longest-running hop.
• ▹Which tools have side effects? Label each tool read-only or side-effecting, then check whether code enforces the difference or whether it only lives in the tool description.
• ▹Does the idempotency key survive a re-plan? Get the model to retry the same intent with slightly different arguments and see whether the key stays the same.
• ▹What does the model see when something fails? Print the exact tool_result content for a timeout, a 429 and a 400. If there's a stack trace in there, fix that first.
• ▹What's the total retry budget per task? Multiply the retry counts across every layer and compare the result with the deadline and dollar limit you'd actually accept for one task.

Bridge to Day 25

Once retries are safe, you need to be able to see them. In the hook trace, the only clue was a 41-second commit in another service's log. Tomorrow is tracing and observability for agent runs: spans across model calls and tool calls, task_id and idempotency keys carried as trace attributes, and a setup where "two refunds, one task" shows up on a dashboard before it shows up in a customer email.

Extend your knowledge

• ▹Marc Brooker, "Timeouts, retries, and backoff with jitter" (Amazon Builders' Library). The standard reference for Layer A. It covers retry amplification, jittered backoff and why retries belong at a single layer.
• ▹Stripe's API docs on idempotent requests, plus Brandur Leach's post "Implementing Stripe-like Idempotency Keys in Postgres". Read these before building your own ledger for tools whose downstream doesn't support keys.
• ▹The gRPC documentation on deadlines and deadline propagation. Model your agent's Budget object on it.
• ▹Temporal's documentation on activities, retry policies and durable execution. It's a production-grade version of the two-layer split, if you'd rather not hand-roll the ledger.