Your Pipeline Pages Someone Twice a Week — and Nobody Downstream Can Tell the Difference

Day 3: Batch vs Streaming — You're Not Choosing Tech, You're Choosing Where Latency Gets Paid

Nobody actually decides between batch and streaming. Someone just reaches for Kafka because that's what the last company ran, and six months later there's an on-call rotation babysitting a pipeline nobody needed in the first place. Getting this call wrong doesn't just cost money — it costs you a pager.

The pager that finally went quiet

Here's a version of this I've watched play out at more than one company. A streaming pipeline pushes user events through Kafka into a feature store, feeding a recommendation model that re-ranks results in real time. Beautiful on the architecture diagram. It also paged someone roughly twice a week — a consumer lag spike, a Flink job stuck in a checkpoint recovery loop, a schema change that broke deserialization for every downstream consumer at once. Eventually someone asked the question that should've been asked on day one: does recommendation quality actually change if the feature is 15 minutes stale instead of 15 seconds stale? It doesn't. Swap in an hourly batch job, and the pager goes quiet. Nobody downstream notices.

The tell nobody checks before reaching for Kafka

Before you touch a streaming system, ask exactly one question: can the downstream consumer even perceive the latency win? That question matters more now than it did ten years ago, because 'downstream consumer' increasingly means an LLM agent, not a human refreshing a dashboard. An agent that only reads your knowledge base when a user asks a question — pulling context at query time via RAG — genuinely cannot tell whether the underlying document was ingested and embedded 10 seconds ago or 10 minutes ago. It doesn't perceive ingest latency at all. It perceives retrieval latency, which is a different problem (Day 4's territory). Streaming ingestion buys you nothing there except a bigger bill and more ways to get paged.

Now contrast that with an agent running a live decision loop — a fraud-check agent scoring a transaction as it happens, a multi-agent trading system reacting to a price feed. There, a 15-minute-stale signal isn't just unhelpful, it's actively wrong. That's a genuine streaming use case. The tell: does the consumer act on the data the instant it changes, or does it just read it whenever it happens to look?

Reframe: it's not a maturity ladder, it's a latency budget

The industry likes to tell this as a maturity story — you start with cron jobs and 'graduate' to Kafka as you scale. Throw that framing out. Batch and streaming are two ends of the same trade: somebody has to wait for the data to be correct and complete. The only real question is who.

• ▹Batch: ingest waits (data accumulates for a window, then processes) — query is instant, because the work is already done by the time someone asks
• ▹Streaming: ingest is instant (data is processed the moment it arrives) — but you've just moved the complexity into keeping that pipeline correct under partial failure, out-of-order events, and load spikes, all in real time

Why 'real-time' wins arguments it shouldn't

Streaming has a demo-impressiveness bias baked into it. Put a live-updating dashboard next to one that refreshes hourly, and the live one wins the room every single time — even if nobody in that room checks it more than once a day. The people who benefit from that impression usually aren't the ones consuming the data. It's the engineer pitching the architecture (streaming reads better on a resume, and in a design doc) and the leader who wants to say 'real-time' in a board update. In the AI era, this bias just wears a new costume: teams build 'real-time agent pipelines' because agentic-plus-real-time sounds cutting edge, while the actual bottleneck is LLM inference latency — hundreds of milliseconds to several seconds per call — which dwarfs whatever few seconds you shaved off ingest. You optimized the part nobody was waiting on.

The bill nobody itemizes

Streaming's operational tax never shows up in the architecture diagram. It shows up in the incident channel. Once you're in production, here's what you're actually signing up to pay for:

• ▹Exactly-once semantics — guaranteeing every event is processed once, not zero or twice, across retries and failures; this is genuinely hard, and most 'exactly-once' claims are really 'effectively-once' with a lot of fine print (idempotency keys, transactional producers)
• ▹Backpressure — what happens when your consumer can't keep up with the producer; without it you either drop data silently or OOM the consumer, and tuning it under real traffic is trial by fire
• ▹Checkpoint recovery — when a stream processor (Flink, Spark Structured Streaming) crashes, it has to resume from a checkpoint without reprocessing or skipping events; get this wrong and you get silent data loss or duplication
• ▹Schema evolution — a producer changes an event shape and every downstream consumer breaks simultaneously, in production, at 2am, because there's no batch window to catch it in staging first
• ▹Consumer lag monitoring — a whole extra category of dashboards and alerts you now own, just to find out if the 'real-time' system is actually keeping up

A 3-question gut check before you pick streaming

• ▹Who's the consumer? A human glancing at a dashboard, a nightly batch job, or an agent making an autonomous decision in the moment?
• ▹What do they do the instant the data lands? If the honest answer is 'nothing, it sits until someone or something looks at it,' streaming buys you nothing.
• ▹What breaks if it's late by 15 minutes? If the answer is 'a slightly staler recommendation' or 'a report that runs at 9:05 instead of 9:00,' you don't have a streaming problem. If the answer is 'we approve a fraudulent transaction' or 'an agent acts on a price that no longer exists,' you do.

needs_streaming =
    consumer.acts_autonomously_on_arrival()
    and consumer.tolerance_window < minutes(1)
    and cost_of_staleness > cost_of(exactly_once + backpressure + checkpointing)

# If this evaluates false, you want batch (or micro-batch) —
# not because it's simpler to build, but because nobody downstream
# can tell the difference, so the ops tax buys nothing.

Where this connects tomorrow

Batch vs streaming only sets the budget for how fresh your data is when it lands. It says nothing about how fast you can find the right piece of it once it's there. Different problem, different tools — and that second one, retrieval, is where Day 4 picks up.

Extend your knowledge

• ▹Read the Kafka documentation on exactly-once semantics (transactional producers, idempotent writes) and compare it against what your own pipeline actually guarantees
• ▹Look up Flink's checkpointing model to see concretely what 'recovery' means when a stream job crashes mid-processing
• ▹Martin Kleppmann's 'Designing Data-Intensive Applications' — the chapter on stream processing is the clearest non-hype treatment of this trade-off out there
• ▹Pull up your own team's consumer lag dashboard, if one exists, and check: has anyone actually looked at it in the last week? If not, ask why the pipeline is still streaming