The Napkin Math That Predicts Whether Your Agent Pipeline Will Crawl

Day 1: Back-of-the-Envelope Estimation

Here's a bet I'll make right now: the next design doc you read will have a number in it that nobody actually checked. "Support 1M requests/day" gets typed with total confidence, and then everyone in the room nods and moves on to arguing about frameworks. Almost every distributed systems decision you'll make in this course — replica count, shard count, whether you can afford the consensus tax — is really a bet on rough numbers dressed up as an engineering decision. If you can't estimate, you can't actually tell a good design from a bad one. You can only tell a confident one from a nervous one, and confidence is not the same thing as correct.

The two-engineers-one-napkin moment

Picture two engineers handed the same doc: "1M requests/day, p99 under 100ms." One opens a spreadsheet, models hourly traffic buckets, and comes back 40 minutes later with a number. The other scribbles on a napkin for 90 seconds and lands on the same order of magnitude. Both of them are right. The spreadsheet engineer just spent 38 minutes buying precision the question didn't ask for. At design-review time you don't need the exact server count — you need to know whether the answer is 3 or 3,000, because that's the difference between tweaking a config value and rewriting the service.

Why this is Day 1, not Day 10

Every topic later in this course is a back-of-the-envelope calculation wearing a nicer outfit. Replication factor is a BOTE calculation about failure probability and read throughput. Partition count is a BOTE calculation about data volume per node. Consensus overhead — Raft, Paxos, all of it — is a BOTE calculation about how many round trips a write can survive before it feels slow. Skip this skill and every later lesson turns into a rule you memorize instead of a number you can derive and, when needed, argue with. You won't be able to open a design doc and ask "wait, does that number actually hold up?" — you'll just trust whoever sounds sure of themselves. That's a bad habit in any team, and it's a genuinely dangerous one on an AI-assisted team, because the confident voice in the room is increasingly an agent, not a senior engineer — and agents are remarkably good at sounding certain about numbers they made up on the spot.

The only numbers worth memorizing: the latency ladder

Don't memorize forty numbers. Memorize one ladder, and pay attention to the gaps between its rungs — because the gaps are what survive as hardware gets faster and the absolute numbers keep moving. This is basically Jeff Dean's old "numbers every programmer should know" table, with one new rung the 2015 version never needed.

• ▹L1 cache reference: ~1 nanosecond — the baseline everything else gets measured against
• ▹L2 cache reference: ~10 nanoseconds — about 10x slower than L1
• ▹Main memory reference: ~100 nanoseconds — about 10x slower than L2, ~100x slower than L1
• ▹SSD random read: ~100 microseconds — about 1,000x slower than memory
• ▹Same-datacenter RPC (service to service): ~0.5-1 millisecond — about 10x slower than an SSD read
• ▹Spinning disk seek: ~10 milliseconds — mostly a museum piece now, but it's why anything touching cold storage still feels sluggish
• ▹LLM inference, one token: ~20-50 milliseconds — the rung nobody had a decade ago, and it quietly rewrites every capacity conversation you'll have from here on
• ▹Cross-region RPC (say, us-east to eu-west): ~100-150 milliseconds — roughly 100-150x slower than staying in one datacenter

Look at where the LLM row sits — between a disk seek and a cross-region call. Now remember you almost never generate one token; you generate hundreds. A single agent call producing a 300-token response is doing the latency-equivalent of several cross-region round trips before your application code even starts running. Chain three agents — planner, tool-caller, summarizer — and you've stacked that cost three times over. This is why multi-agent pipelines feel sluggish even when every individual call technically "succeeds fast": you're not fighting bugs, you're fighting the ladder itself. The millisecond figures will keep shrinking as models improve — the ratios between rungs are the part worth carrying in your head, because they tell you where the next 10x win is actually hiding. Hint: it's in batching more tokens per call, not in shaving microseconds off your JSON parser.

Worked example: how many servers for 1M requests/day at p99 < 100ms?

"1M requests/day" sounds like a lot. It isn't — not until you nail down four things, and it's these four assumptions, not the headline number, that actually decide your answer.

• ▹Peak-to-average ratio — traffic is never flat; assume peak is ~5x the daily average, which is reasonable for a consumer app with a daytime usage curve
• ▹Time per request — how long the server actually holds the request open (say, 50ms of real work)
• ▹Concurrency per server — how many requests one machine can juggle at once before p99 starts to sag (this depends on whether the work is CPU-bound or I/O-bound — and for AI workloads, on GPU batch size)
• ▹Target utilization — you never run at 100% capacity; running at 30-50% is what protects your p99, not your average

1. Average load
   1,000,000 req/day / 86,400 s/day ≈ 12 req/s

2. Peak load (assume 5x average)
   12 req/s * 5 ≈ 60 req/s

3. Per-server theoretical capacity
   assume 50ms/request, 20 concurrent in-flight requests (I/O-bound, async)
   capacity = concurrency / latency = 20 / 0.05s = 400 req/s

4. Effective capacity at safe utilization (40%, to protect p99)
   400 req/s * 0.4 ≈ 160 req/s per server

5. Servers needed
   60 req/s (peak) / 160 req/s (per server) ≈ 0.4 → round up to 1
   add redundancy for failover (N+1 or N+2) → 2-3 servers

That's the whole answer: a system described in a design doc as if it needs a fleet often needs two or three boxes plus redundancy. Now swap out assumptions #2 and #3 for an AI-backed version of the same endpoint — each request now makes one LLM call (2 seconds, not 50ms), and each GPU instance can only batch about 8 concurrent generations before latency degrades. Rerun the exact same math and that same 1M requests/day suddenly needs dozens of GPU instances, not three CPU boxes. Same formula, same four assumptions — the traffic number never moved, but the answer jumped 10x because the *type* of work changed underneath it. That's the entire point of this skill: the headline number is almost never what decides your architecture.

The rule most people get wrong: round to the nearest power of 10

Rounding aggressively feels like cheating, especially if you were trained to respect precision. At this stage it's the opposite of cheating — computing "163.7 req/s per server" to one decimal place is fake confidence, because your inputs (peak ratio, request time, concurrency) were guesses to begin with. If your guesses are off by ±50%, your third decimal place isn't rigor, it's noise wearing a lab coat. Round every input to the nearest power of 10 (or a nice number: 1, 3, 10, 30, 100...), do the arithmetic in your head, and only open a spreadsheet once the rough answer tells you which regime you're in. This matters even more when you're pricing out an agent pipeline — token counts, tool-call counts, per-call latency should all get bucketed to an order of magnitude before you do anything else. Whether a multi-agent fan-out costs $0.02 or $0.024 per request never changes a decision. Whether it costs $0.02 or $0.20 always does.

A 5-minute drill: estimate a chat app's storage growth per year

Pick an app you use every day — chat apps are the cleanest example for this. Before you read the next paragraph, actually try it: how many daily active users, how many messages per user per day, how many bytes per message including metadata, and what does that add up to over a year? Then check your reasoning against the version below.

Assumptions (rounded, power-of-10 style):
  - 100M daily active users
  - 20 messages/user/day
  - 100 bytes/message (text) x 5 (overhead: sender id, timestamp, indexes, replication) = 500 bytes effective

Messages/day = 100,000,000 * 20 = 2 x 10^9
Bytes/day    = 2x10^9 * 500 bytes = 10^12 bytes = ~1 TB/day
Bytes/year   = 1 TB/day * 365 ≈ 3.6 x 10^2 TB/year

Rounded to nearest power of 10 for capacity planning: ~1 PB/year

If your app is smaller, scale the DAU number down and redo it — the method doesn't change one bit. The lesson isn't "chat apps burn a petabyte a year." It's that four honest assumptions, multiplied out and rounded without mercy, get you within striking distance of a real number in under five minutes — which is exactly the estimate you'd want in hand before anyone starts arguing about sharding strategy.

Bridge to Day 2

Every calculation today quietly assumed one thing: that a single machine, once provisioned, just keeps running. Tomorrow we break that assumption — and every number you calculated today changes shape the moment you have to plan for a machine dying mid-request.

Extend your knowledge

• ▹Look up 'Latency Numbers Every Programmer Should Know' (originally attributed to Jeff Dean at Google) and its interactive visualizations — the canonical reference table for the latency ladder.
• ▹Read Chapter 1 of Martin Kleppmann's 'Designing Data-Intensive Applications' for the deeper treatment of percentiles (p50/p99) and why averages mislead you.
• ▹Google's 'Site Reliability Engineering' book covers capacity planning and overload handling with the same peak-ratio and utilization-margin reasoning used in today's worked example.
• ▹This week: instrument one real endpoint (ideally one that calls an LLM) and log its actual p50/p99 latency, then compare it to what you'd have estimated cold — the gap is the most useful thing you'll learn this week.