Streaming vs batching in agent systems

Day 21: Stream to Humans, Batch Between Agents

If your app streams tokens and also runs a verifier, you've built a fact-checker for text people have already read. The check runs, it catches the error, and the user is already acting on the wrong answer. This lesson covers where streaming actually belongs in an agentic system, so your checks still protect the user and your agents never act on half-finished output.

The answer the verifier rejected two seconds too late

I've hit this pattern myself, and I suspect most teams hit it soon after they add a critic. A support assistant streams an answer about a refund policy, token by token. It feels fast. A verifier runs once generation finishes, checks the answer against the policy document, and finds a wrong number. The check completes about two seconds after the last token appeared on screen. By then the user has read the answer, and they may already have pasted it into an email.

Every option from there was bad. We could swap the message for a corrected one, which looks like the system contradicting itself. We could append a correction, which looks worse. Someone in the room suggested "just add a disclaimer". That fixes nothing. A disclaimer doesn't change what the user read or did, people stop seeing disclaimers almost immediately, and it makes users trust your correct answers less too. The real problem was in the architecture: we published the output before we checked it.

The reframe: streaming is early commitment

Every token you send out is a promise you can't take back. Once it's on a screen, in a log the user can see, or in another agent's context, you can't verify it, retry it or roll it back.

• ▹What streaming buys you is perceived latency. Time-to-first-token is usually a fraction of the full generation time, often a few hundred milliseconds instead of several seconds, so the user sees something happening.
• ▹What it costs you is verifiability. Anything you stream skips every check that runs after generation: validators, critics, policy filters, schema checks.
• ▹So the trade only makes sense when nothing after generation is worth waiting for. Add a verifier and you're paying the cost while throwing away the safety the verifier was meant to give you.

Callback: why this breaks Day 20's verifier loop

On Day 20 you built a generate → critique → revise loop. The whole loop depends on one assumption: the draft stays uncommitted until it passes. The critic rejects draft one, the reviser writes draft two, and nobody outside the loop ever sees draft one. Stream the generator's tokens to the user and that assumption is gone. They watch draft one being written. When the critic rejects it, you aren't revising a draft any more. You're retracting something you already published. The code still runs, so the loop looks healthy, but it has quietly stopped doing its job.

The bug factory between agents

Between agents it gets worse. A confused human just reads the wrong thing. A downstream agent acts on it. Model APIs stream tool-call arguments as partial JSON fragments: Anthropic sends input_json_delta events, and OpenAI sends function-argument deltas. It's tempting to start work early by feeding those fragments into a lenient JSON parser and dispatching the moment the parse "succeeds". Here's what that looks like when it goes wrong:

Agent A (streaming tool args):
  {"action": "refund", "order_id": "A-1029", "amount": 4
                                                       ^ stream still going (49.99 coming)

Lenient partial-JSON parser:
  {"action": "refund", "order_id": "A-1029", "amount": 4}   <- parses fine

Agent B / executor:
  refund(order_id="A-1029", amount=4)   <- fired. Real money. Wrong amount.

This is speculative execution without rollback. CPUs speculate constantly, but they can throw the work away when the guess turns out wrong. Your refund API can't. Neither can your email sender or your database write. Plans fail the same way: agent B starts step 1 of agent A's plan while A is still writing step 3, and step 3 says "actually, skip step 1".

The fix is boring, and that's fine. Act only on complete units. For tool calls, wait for the block-complete event (content_block_stop on Anthropic, the arguments-done event or finished tool call on OpenAI). Then parse strictly, validate against the schema, and only then dispatch.

The rule: stream to humans, batch between agents

Inside the pipeline, buffer everything: whole messages, whole tool calls, whole plans. Streaming happens only at the edge, and only after the last gate.

• ▹Generator → buffer: collect the full output. Nothing leaves yet.
• ▹Buffer → verifier: strict parsing, schema checks, critic and policy checks all run on the complete artifact.
• ▹Commit point: the one place where output becomes visible or actionable. Give it a name in your code.
• ▹Edge: only committed output reaches the user. The dotted lines are progress events. Those are safe to stream because they report things that have already happened.

Users still need to see movement, so change what you stream

Buffering the answer doesn't mean staring at a spinner for 15 seconds. Stop streaming answer tokens and stream state transitions instead: planning, calling search(…), got 12 results, verifying, done. Each one is true the moment you send it, so there's never anything to take back. The user sees real progress, and none of it is unverified content.

// Minimal progress event schema
type AgentEvent = {
  run_id: string;      // one per user request
  seq: number;         // monotonic, lets the client order and dedupe
  ts: string;          // ISO timestamp
} & (
  | { type: "phase"; name: "planning" | "drafting" | "verifying" | "done" }
  | { type: "tool_call"; tool: string; args_preview: string }   // redacted, human-readable
  | { type: "tool_result"; tool: string; summary: string }      // "12 results", not raw output
  | { type: "decision"; action: "revise" | "retry" | "stop"; reason: string }
  | { type: "answer"; text: string }                            // only after the commit point
  | { type: "error"; message: string }
);

Here's a server sketch using Server-Sent Events (FastAPI). Progress goes out immediately. The real answer stays in memory until it passes verification. plan_steps, execute, generate_answer, verify and revise are placeholders for your Day 20 pipeline. To keep it short, the sketch skips run_id, seq and ts. Add them in your helper, because Day 22 needs them.

import json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

def sse(event: str, data: dict) -> str:
    return f"event: {event}\ndata: {json.dumps(data)}\n\n"

@app.get("/ask")
async def ask(q: str):
    async def run():
        yield sse("phase", {"name": "planning"})
        plan = await plan_steps(q)                    # buffered, never streamed raw
        for step in plan.tool_calls:                  # complete, schema-validated calls only
            yield sse("tool_call", {"tool": step.name, "args_preview": step.preview()})
            result = await execute(step)
            yield sse("tool_result", {"tool": step.name, "summary": result.summary})
        yield sse("phase", {"name": "drafting"})
        draft = await generate_answer(q, plan)        # full answer held in memory
        for attempt in range(2):                      # cap retries
            yield sse("phase", {"name": "verifying"})
            verdict = await verify(draft)
            if verdict.ok:
                yield sse("answer", {"text": draft})  # <- the commit point
                break
            yield sse("decision", {"action": "revise", "reason": verdict.reason})
            draft = await revise(draft, verdict)
        else:
            yield sse("error", {"message": "Couldn't produce a verified answer."})
        yield sse("phase", {"name": "done"})
    return StreamingResponse(run(), media_type="text/event-stream")

• ▹Like the typewriter feel? Once the answer is committed, the client can still reveal it character by character. That's a display choice and nothing more. The content was verified before it left the server.
• ▹Use args_preview, not raw args. Progress events reach the user too, so strip personal data, secrets and internal IDs before you emit them.
• ▹Keep intent as short structured labels. Streaming the model's raw reasoning and calling it "progress" just smuggles unverified output back in under a new name.

When streaming raw tokens is still fine

Raw token streaming isn't always wrong. It's the right default for plain chat with no downstream check, for drafts the user expects to edit (emails, code they'll review), and for low-stakes summaries. Before you stream raw tokens anywhere, ask three questions:

• ▹1. Does anything verify the output after generation? If yes, buffer. Otherwise the verifier is just for show.
• ▹2. Does anything act on this output (a tool, another agent, a database write, a send)? If yes, batch it and act only on complete, validated units.
• ▹3. Would a retraction hurt trust? If showing an answer and then pulling it back would confuse or mislead the user, buffer.
• ▹Three noes? Stream raw tokens and enjoy the latency win.

A side effect we didn't expect: better interruptions

Once users saw intent and tool calls instead of scrolling text, they interrupted differently. With raw token streams, people hit stop mid-paragraph because one sentence looked off, and often killed a run that would have turned out fine. With progress events, they stopped at the moment it actually mattered. At step 3 they'd see calling search("refund policy 2019"), realise the query was wrong, and fix it before a single answer token was built on bad results. Step boundaries also give your cancel button a clean place to land: you abort between steps, not halfway through a tool call.

Checklist to apply today

• ▹Find every place where output crosses a boundary: model → UI, model → tool, agent → agent, agent → queue or database.
• ▹Label each one as human-edge or agent-internal.
• ▹Buffer every agent-internal boundary. Complete messages and complete tool calls only, strictly parsed and schema-validated before anything acts on them.
• ▹Put explicit commit points in your code (one function, one log line) where output becomes visible or actionable.
• ▹At the human edge, stream progress events (phase, tool_call, tool_result, decision). Send the answer only after it clears the last gate.
• ▹For any surface that still streams raw tokens, run the three-question test and write down why it's allowed to.

Bridge to Day 22

Look at what you've just built: events with a run_id, a sequence number and timestamps, tool calls with arguments and results, decisions with reasons. That's most of an observability trace already. Tomorrow we treat it as one, and turn these events into spans you can search, replay and alert on.

Extend your knowledge

• ▹Read the Anthropic Messages API streaming docs, especially input_json_delta and content_block_stop, so you know exactly when a tool call is complete enough to act on.
• ▹MDN's 'Using server-sent events' guide covers event names, event IDs and reconnection (the Last-Event-ID header). It pairs nicely with the seq field in your event schema.
• ▹Before Day 22, skim the OpenTelemetry GenAI semantic conventions. Your progress events map closely onto their spans and attributes.
• ▹Exercise: take your Day 20 verifier loop, put a named commit_answer() function at the commit point, and add the SSE progress endpoint from this lesson. Then confirm nothing reaches the client before commit_answer() runs.