Posts tagged Latency — AI × software engineering, made practical.
You'll learn how prompt caching makes time-to-first-token depend on the order of your prompt, how to find the lines that break the cache in your agent harness, and how to order the prompt so the cache keeps hitting on every turn.
We went from 4 subagents to 16 expecting more throughput — p95 latency got worse instead, and the culprit turned out to be the same failure mode database engineers have fought for decades.
A gut-check for engineers who reach for Kafka by default: if the consumer can't perceive the latency win, streaming isn't buying speed, it's buying you an incident channel.
One back-of-envelope habit tells you, before you write a line of code, whether your design needs three servers or three hundred — and why your multi-agent pipeline is already slow for reasons that have nothing to do with your code.
Your dashboards say every node is healthy, but one of them is quietly swallowing every expensive tool-use chain — because round-robin and least-connections were never built to know what an LLM request actually costs.