The Spreadsheet That Quietly Beat My 'Smarter' AI Feature

Day 8: The Workaround Is the Interview You Skipped

Somewhere right now, a user of yours has a spreadsheet open in another tab — one you didn't build, that quietly does the job your feature was supposed to do. Find that spreadsheet and read it honestly, and you've just done more user research than a year of surveys, because it's not an opinion. It's a behavior.

The screen share

On a support call, a customer shares their screen to show me a bug. Before I even look at it, they tab past a spreadsheet — just for a second, but I catch it. It's a manual rebuild of the AI triage feature I'd shipped six months earlier, the one that used an LLM to rank incoming requests by urgency. Their version has a column of hand-typed scores, a few formulas, and notes justifying every single one. It's slower than my feature. It's also, unmistakably, better — because they trust every number in it.

Three months of denial

I saw that spreadsheet on the call. Then I sat on it for three months. Every time it resurfaced, I had an excuse ready — and each one was really just a way of not looking straight at what the spreadsheet was telling me:

• ▹"They just haven't been onboarded right" — puts the blame on the user's understanding instead of on what the product actually outputs.
• ▹"Power users always do this" — turns a warning sign into a compliment so you never have to dig into it.
• ▹"It's an edge case" — assumes the workaround is rare, without ever checking how many other customers were quietly running the same one.
• ▹"The model just needs better prompts, more fine-tuning" — the AI-era version of the same dodge: recasts a trust problem as a model-quality problem.

Every one of those excuses let me go on believing the feature worked. Not one of them required me to actually open the spreadsheet and count columns.

The audit I finally forced myself to do

Eventually I put the spreadsheet and my product side by side, column by column. Theirs had: the raw signal, the score, a one-line reason, and a manual override column. Mine had: a score. That's it — no reason attached, no way to see what the model had weighted, no way to correct a bad call without filing a ticket. The spreadsheet won on every column that actually mattered when someone had to defend a decision to their own boss.

The uncomfortable finding: a trust gap, not a feature gap

I'd assumed the spreadsheet existed because my feature was missing some capability. It wasn't. The spreadsheet was legible — you could see exactly why a number was what it was — and it was editable — you could fix it on the spot the moment it was wrong. My 'smarter' AI feature was neither of those things. This is the pattern to watch for in any AI product right now: an agent or a model can be more accurate on average and still lose to a dumber tool, because accuracy was never what earned the trust. Inspectability and correctability are what earn it. A user who can't see the reasoning behind an agent's call, and can't override it without opening a ticket, will quietly go rebuild that call somewhere they control.

Why I couldn't see it sooner

Sunk cost isn't only about the hours you put in — it's about identity. I'd built part of my internal reputation on that AI scoring feature. It was the thing people pointed to when they said I actually knew how to ship 'real AI, not just a wrapper around a prompt.' Admitting the spreadsheet was better meant admitting that the thing I was known for was the exact thing people were routing around. That's a much harder sentence to say out loud than 'we missed a feature' — which is exactly why engineers, especially ones proud of an agent or model feature they shipped, are the last people to notice the workaround forming around it.

The reusable technique

Stop asking users 'what do you want the product to do.' You'll get back feature requests, and those requests are bounded by whatever the user already believes is possible. Ask this instead: 'Walk me through everything you touch, in order, right before and right after you open my product.' The workaround always lives in that gap — the exported CSV they clean up first, the ChatGPT tab they paste your output into to double-check it, the manual override log they keep because they don't actually trust the agent's decision. You're not asking them to evaluate your product. You're asking them to narrate their day, and the shadow tool falls straight out of that narration without them ever having to frame it as a complaint.

• ▹In an AI product specifically, look for: a parallel prompt library, a manual review pass bolted onto the agent's output, an exported log kept 'just in case,' or a spreadsheet that reconstructs the reasoning behind a score your model hands back as one opaque number.
• ▹Every one of those is a workaround. None of them will surface if the only question you ask is 'do you like the AI feature.'

Day 8 takeaway

User research isn't a meeting you put on the calendar — it's an eye for the tools people quietly build so they don't have to ask you for help. You'll never find the workaround by accident, because finding it means admitting the thing you built is the thing being avoided. Tomorrow, Day 9: what to actually do once you've found your spreadsheet.

Extend your knowledge

• ▹Read up on 'shadow IT' in enterprise software research — the same pattern, unauthorized tools filling a trust or capability gap, has been studied for decades outside of AI, and the diagnostic questions transfer over directly.
• ▹Look at Teresa Torres's continuous discovery habits (opportunity solution trees) for a way to keep asking the before/after walkthrough question on a regular cadence, instead of only once a crisis forces it.
• ▹If you ship agent or LLM features, audit your own product this week: can a user see the reasoning behind any AI-generated decision? Can they correct it without filing a ticket? If not, assume a spreadsheet is already forming somewhere.
• ▹Check your product analytics or support logs for exports, copy events, or repeated manual overrides right after an AI feature fires — that's the workaround leaving a paper trail before anyone's told you it exists.