Checking confident AI answers

Day 6: Stop listening to how the AI sounds. Check what it claims.

Your assistant uses the same voice when it's right and when it's making things up. It has the same calm tone, the same tidy formatting and the same "just add this line." So the tone tells you nothing. If you can't separate facts from guesses, sooner or later you'll act on a guess. The more specific the guess, the more it usually costs you.

Which one is fake?

Someone asks an assistant: 'How do I make sure my Python requests calls don't hang forever?' Here are two answers. Both sound calm and certain.

Answer A
The requests library does not apply a timeout by default, so a call can
wait indefinitely. Pass timeout= on every call, for example
requests.get(url, timeout=10), or wrap requests in a small helper or
Session subclass that always sets one.

Answer B
Since requests 2.32 you can set a global default once at startup:
requests.defaults.timeout = 30. Every call after that inherits it, and
you can still override it per call with timeout=.

B is the fake. I wrote it for this lesson, and as far as I know requests has no `requests.defaults.timeout` setting. A matches the library's documentation. Now reread them. Nothing in B's wording gives it away. If anything, B is more convincing, because it's more precise: it names a version, an attribute path and a default value. That precision is exactly what should make you suspicious.

A small moment from my own week

I'm not immune to this. I asked an assistant how to change a timeout in a tool we use. It gave me a config key with a tidy, sensible-looking name, and I pasted it in without a second thought. The service started. Nothing complained. The behaviour just didn't change. I spent longer than I'd like to admit looking for the problem in the network, then the deployment, then my own code. When I finally opened the tool's actual docs, the key wasn't there. The config loader had been quietly ignoring an option that never existed. The answer wasn't vague. It was exact, and that's why I trusted it.

Why this happens, in plain words

A language model writes by predicting the most plausible next words given everything before them. Nothing in that process checks whether a detail is real. `client_max_body_size` (a real nginx setting) and `proxy_upload_buffer_limit` (a made-up one) are both plausible strings of words. The model produces either one with the same effort and the same fluency.

• ▹Confidence is a style, not a measurement. The model learned from text written by confident experts, so it sounds like one whether or not it knows the answer.
• ▹There's no built-in 'I'm guessing' voice. Newer models do say 'I'm not sure' more often, but you can't rely on it, and they almost never say it about small details like a flag name.
• ▹Agents raise the stakes. Once an assistant can edit files, run commands or call APIs, it doesn't just say the invented detail. It acts on it, and the failure can stay quiet for a long time.

Think of a very fluent new colleague who never says 'I don't know.' They're often right. They explain things well, and they're fast. But when they hit the edge of what they know, their voice doesn't change at all. You wouldn't fire them. You'd learn to ask, 'Where did you get that?'

The flip: specific means check first

With people, we run a simple rule: vague means unsure, specific means they know. A colleague who quotes the exact version number has usually looked it up. With models, flip that rule for anything without a source. The more specific the claim, the more it needs one. Vague advice is hard to get badly wrong. An exact detail is easy to invent and easy to act on.

TRUST GRADIENT (for claims with no visible source)

Low risk, fine to take as a starting point
  - General approach: 'add a timeout', 'cache the result', 'write a test first'
  - Trade-off explanations: 'X is simpler, Y scales better'
  - Restating things you already gave it

Medium risk, sanity-check before relying on it
  - 'Most teams do X', 'this is the usual pattern'
  - Rough sizes and orders of magnitude

High risk, check BEFORE acting
  - Exact numbers: limits, defaults, prices, percentages, dates
  - Names: functions, flags, config keys, env vars, CLI options, API fields
  - Versions: 'added in 2.32', 'deprecated since v5'
  - Paths and URLs: file locations, endpoints, doc links
  - Quotes and citations: papers, law sections, policy clauses, 'as X said'

Technique 1: The strip-the-tone pass

Take the answer and rewrite it as a plain list of claims. Cut every word that adds confidence but no information: 'definitely', 'simply', 'just', 'as you know', 'the standard way', 'this will fix it'. Once the tone is gone, what's left is a list of claims you can check, and the guesses are much easier to spot.

BEFORE (the answer as it arrived)

This is a common nginx issue and the fix is simple. You're hitting the
default upload limit, which is 1 MB. Just add client_max_body_size 50m;
to your server block. You'll definitely also want
proxy_upload_buffer_limit 50m; since nginx 1.25, otherwise large uploads
are buffered to disk and time out. Reload nginx and you're done. This is
the standard setup for any app accepting file uploads.


AFTER (bare claims, numbered)

1. nginx has a default request body limit of 1 MB.
2. The setting that controls it is client_max_body_size.
3. It can be set in the server block.
4. A setting called proxy_upload_buffer_limit exists.
5. That setting was introduced in nginx 1.25.
6. Without it, large uploads are buffered to disk and time out.
7. A reload applies the change.
8. 'This is the standard setup' -> no claim, delete it.

Claims 1 to 3 and 7 match nginx's real documentation. Claims 4 and 5 are invented. I don't know of any such directive or version note. Claim 6 is the sneaky one because it's half true. nginx really does write request bodies larger than client_body_buffer_size to a temporary file on disk. But that has nothing to do with a missing directive, and it doesn't make uploads time out by itself. The fake borrowed a real detail to look credible. In the original, the fake part was also the most confident sentence ('You'll definitely also want'). In the bare list it sits next to the real claims with nothing holding it up. It also breaks into three separate claims (a name, a version and a cause), and you can check each one on its own.

Technique 2: Where did this token come from?

Now go through every specific item in your stripped list: each number, proper noun, identifier, path and quote. For each one, ask where it came from, and give it one of three tags.

• ▹SEEN: it shows up in something you can point to in this conversation. That could be text you pasted, a file the assistant actually read, a search it actually ran, or a tool's output.
• ▹REASONED: it follows by clear logic from something SEEN, like 'your log shows a 413 error, so the body limit is being hit.' You can check the reasoning yourself.
• ▹NOTHING: you can't see where it came from. Treat it as a hypothesis, not a fact. Don't paste it into config or forward it to a stakeholder until you've confirmed it.

If your assistant uses tools (Claude Code, Copilot agent mode, or ChatGPT and Claude with search on), check the activity log, not the summary. 'I checked the docs and confirmed X' is just more generated text. The evidence is the tool call itself: a visible search with results, a file read showing the lines it read, a command with its output. If the summary says 'I verified' and the log shows no verification step, tag the claim NOTHING.

Technique 3: Push on it

Grounded facts hold up under pressure. Guesses drift or collapse. When a NOTHING claim actually matters, lean on it. Here are three ways.

• ▹Ask again in a fresh chat. Start a new conversation with no history and ask the same question in neutral wording. Don't hint at the answer you got before. If the same specific detail comes back, that's mild reassurance. If the version, number or flag name changes, that's strong evidence it was a guess.
• ▹Ask it to argue the opposite. In the original chat, try: 'Argue why claim 4 is wrong. What would I see if that setting didn't exist?' A grounded answer usually holds its ground and explains why. A guess often flips immediately.
• ▹Watch how fast it agrees. If you push back and it instantly says 'You're right, I apologise' and hands you a different answer, that tells you something too. Training on human feedback tends to make models agreeable (researchers call this sycophancy). A claim it drops at the first nudge was never well supported.

This connects to my own research on multi-agent systems. When you run several agents or samples on the same question, the spots where they disagree are often the best signal of where the system is unsure. That disagreement is the uncertainty the confident tone hides. Asking again in a fresh chat is the one-person version of the same idea.

PUSH PROMPTS YOU CAN COPY

Fresh chat:
  "In nginx, is there a directive for limiting upload buffering separately
   from client_max_body_size? Name it only if you're sure it exists."

Same chat:
  "List every config key, version number and default you gave me above.
   For each, say whether it came from something in this conversation or
   from memory."

  "Argue the strongest case that proxy_upload_buffer_limit does not exist."

Cheat sheet: signs a guess is dressed up as a fact

• ▹An exact number (limit, default, price, percentage, date) with no source in sight.
• ▹A plausible name you've never seen before, like a function, flag, config key, paper title or law section, that nothing in the chat could have supplied.
• ▹Past-tense claims of action ('I verified', 'I tested', 'I checked the docs') with no visible tool call or output behind them.
• ▹Appeals to authority with nothing underneath: 'this is the standard way', 'best practice is', 'everyone uses'.
• ▹Hedging that only appears after you push. If 'it may depend on your version' shows up in the second reply, the first reply was a guess.
• ▹Citations or links you can't open, or that open to something that doesn't say what was claimed.
• ▹Details that shift between fresh chats while the overall advice stays the same.

Where this fits in the series

The earlier days were about input: giving the assistant good context, clear goals and the right material. Today is about output, which means reading what comes back without being swayed by how it sounds. The two are linked. Every piece of context you give the model is something real it can draw on, and that turns NOTHING tokens into SEEN ones. Next time we'll go a step further and get the assistant to show its sources up front, tagging its own claims as it writes, so there's less for you to check afterwards.

Your 10-minute exercise for today

• ▹Pick one AI answer you already acted on this week: a config change, a number in a report, a snippet you merged.
• ▹Strip the tone (3 minutes). Rewrite it as a numbered list of bare claims.
• ▹Tag every specific token (4 minutes). Mark each number, name, version, path and quote as SEEN, REASONED or NOTHING.
• ▹Push on the NOTHINGs (3 minutes). Take the most important one to a fresh chat, or check it in the official docs.
• ▹Count. Write down how many specific claims there were and how many had a real source.

Extend your knowledge

• ▹Read 'Towards Understanding Sycophancy in Language Models' (Sharma et al., Anthropic, 2023). It explains why models cave when you push back, and why quick agreement is itself a signal.
• ▹Skim SelfCheckGPT (Manakul et al., 2023) and 'Self-Consistency Improves Chain of Thought Reasoning' (Wang et al., 2022). They're the research versions of asking again in a fresh chat: sample several answers and treat disagreement as a sign of uncertainty.
• ▹Turn on the tool or activity view in the assistant you use most (Claude Code's tool calls, Copilot agent mode's steps, ChatGPT's search sources) and actually read it until checking the log becomes a habit.
• ▹Turn the cheat sheet into a team checklist, and add one line to your PR template: 'Any AI-suggested config keys, versions or limits verified against official docs?'