Your A/B Test Had a Pre-Written Ending — and Nobody Noticed

Day 7 — The A/B Test That Was Never a Test

Most of the 'A/B tests' your team ran this quarter were rigged. Not on purpose — nobody sat down and decided to fake it. But rigged all the same, because the losing number was never going to change anything. A new prompt, a cheaper model, a different retrieval strategy — you ran the test, you got a chart, and the chart was never actually in charge of the decision.

The dashboard that lied by omission

Picture the retro. Your team swapped the support agent's system prompt for a shorter one, chasing token cost. The dashboard says resolution rate is down 1.8%, CSAT is flat, cost is down 30%. Someone calls it 'within noise.' Someone else says 'but the cost savings are real.' It ships. Three weeks later, in the next retro, someone finally says the quiet part out loud — 'so... are we reverting the prompt?' Silence. Nobody reverts it. Nobody re-runs it. The chart existed. The p-value existed. None of it touched the actual decision, which had already been made the day finance asked about the LLM bill.

Name it: a vibes check with a chart stapled on

This isn't a stats failure. It's the same product-mindset failure from earlier this week — 'what we already decided' wearing the costume of 'what the data says.' Three tells give it away, every single time:

• ▹n=1 cohort — one team's dogfooding, one customer segment, one eval set of 50 hand-picked transcripts. There's no population to generalize from, just a vibe with a sample size attached.
• ▹No real randomization unit — the sneaky one in agentic systems. A 'user' in your dashboard might be one person's 40-turn conversation with your agent. Randomize at the session level while outcomes are correlated turn-to-turn — one bad tool call poisons everything after it — and your 'n=2,000 interactions' is really n=80 independent trials wearing a bigger number's clothes.
• ▹The decision was made before the data existed — the model swap was happening regardless, because of cost, a deadline, or someone senior's preference. The test was commissioned to produce a chart for the decision, not evidence to inform it.

Why engineers fall for it

A p-value feels like objectivity, and objectivity is armor in a roadmap review — nobody can accuse you of shipping on gut feel if you walk in holding a chart. AI-era work makes this especially easy to fall into, because you already have a dashboard for everything: token cost, latency, an LLM-as-judge score, a thumbs-up rate. Pointing at a number after the fact and calling it evidence takes zero extra effort. The chart gets called in as a defense witness for a decision that was already made — never deposed as a source of truth before the decision happened. That's the tell. Evidence collected to justify beats evidence collected to decide, every time, because only one of those was ever allowed to say no.

The one question that separates real from theater

Before you ran this, would you have shipped the loser? Ask it about your last three ship decisions, honestly. If a prompt change had come back with worse resolution and identical cost, would the team have reverted it — really, in practice, with a name attached to who does it? If the honest answer is 'probably not, we'd have found a reason to ship it anyway,' you didn't run a test. You ran a ritual that dresses an existing decision up in statistical language.

What a real minimal-bar experiment requires — for an engineer, not a growth team

You don't need an experimentation platform to run something real. You need three things, and in agentic systems each one has a specific, easy-to-miss way of quietly failing:

• ▹A pre-committed decision rule, written down before launch — 'if resolution rate drops more than 2pts, we revert within 48 hours, no exceptions, no re-litigation' — posted somewhere with a name attached, before anyone has looked at a single data point.
• ▹A randomization unit that actually isolates the variant — for agent and LLM work this almost always means the user or the conversation thread, not the turn or the tool call. Check that your variant assignment isn't leaking — a shared cache, or a routed-to-cheaper-model fallback, bleeding into the control group.
• ▹Enough volume that noise doesn't disguise itself as signal. LLM outputs are noisy by construction — temperature, retrieval variance, the occasional tool failure — so a 'test' comparing two prompts across 200 conversations is mostly measuring sampling noise. Know your minimum detectable effect before you look at the dashboard, not after.

Same muscle, new ritual

This is the identical skill you've been building all week — separating what the data says from what you'd already decided. Earlier in the series that showed up as knowing your metric before you needed a good story about it. Today it shows up wearing a specific costume: the A/B test, the ritual engineers trust more than a gut call precisely because it looks rigorous. The rigor is fake if the willingness to lose was never real. An eval score, an LLM-judge win rate, a cost dashboard — none of these are experiments until someone was actually prepared to be told no by them.

The test to run on your own team

Next time your team ships an 'A/B test,' find whoever ran it and ask one question: 'if this had come back negative, what happens next, and who does it?' A clear, specific answer — a name, a date, a revert PR pre-written and sitting there — means you ran an experiment. A shrug or a 'well, it depends' means you ran a chart for a decision that was never up for debate. Say that plainly in the retro. That's the product-mindset move here — not blocking the ship, just refusing to let the chart pretend it did work it never did.

Extend your knowledge

• ▹Audit your last three shipped 'A/B tests': for each, find the written decision rule (if one exists) and check if it was written before or after the first data point.
• ▹If your team evaluates LLM/agent changes with an LLM-as-judge or eval-set score, check what your randomization/sampling unit actually is — conversation, turn, or user — and whether that matches what the dashboard implies.
• ▹Read up on 'pre-registration' from clinical trial methodology — it's the same discipline (write the analysis plan before you see the data) applied outside software, and it's where this idea is most rigorously enforced.
• ▹Next experiment you run, write the revert criteria in the PR description before merging, with an owner's name attached — make the pre-commitment artifact-visible, not just a memory.