Hallucinated function names

Day 8: Why AI invents function names, and the 10-second check that catches it

The AI code that hurts you isn't the code that looks wrong. It's the line that looks so right you never think to check it: the kwarg that should exist, or the flag every similar tool has. Learn where those fakes tend to hide and run one quick check there, and most of them won't reach your codebase.

Hook: which line would you paste?

Both lines below try to drop duplicate rows by `id` and keep the latest one. One is real pandas. The other doesn't exist. Pick one before you read on.

# A
df = df.drop_duplicates(columns=["id"], keep="last")

# B
df = df.drop_duplicates(subset=["id"], keep="last")

Most people I show this to pick A. It reads better, and it mirrors `df.drop(columns=[...])`, which you've probably typed more times than you can count. But A is the fake. `drop_duplicates` takes `subset=`, so A fails with `TypeError: ... got an unexpected keyword argument 'columns'`. The line that felt more natural is the invented one. That one fact is today's lesson.

The mechanism in one sentence

The model predicts the next piece of text. Unless you've given it a tool to search docs or inspect your environment, it isn't looking anything up. It's the same next-token mechanism from earlier in the series, applied to API names. The model has seen a huge number of libraries with `keep='last'`, `retries=`, `--dry-run` and `--force`. So when it writes a call, it writes the most likely-looking version of that call. That's roughly an average of all those libraries, which isn't necessarily the API you actually have installed.

Why fakes look so convincing: they follow the conventions

A hallucinated symbol is usually the one that fits the pattern. It turns up exactly where the real API breaks its own ecosystem's conventions. Three common cases:

# 1. pandas kwarg borrowed from a sibling method
df.drop_duplicates(columns=["id"])      # fake: drop() uses columns=, drop_duplicates() uses subset=

# 2. A retry option that "should" exist
requests.get(url, timeout=5, retries=3)  # fake: TypeError, unexpected keyword 'retries'
# Real way: mount HTTPAdapter(max_retries=urllib3.util.Retry(...)) on a requests.Session

# 3. CLI flag borrowed from a sibling tool
terraform apply --dry-run
# kubectl and helm have --dry-run, so it looks plausible.
# Terraform's equivalent is a separate command: terraform plan

Fakes tend to cluster in four places:

• ▹Inconsistent APIs, like a library that uses `subset=` in one method and `columns=` in another.
• ▹Anything that changed recently: renamed or removed parameters, or a major version bump (pandas 1 → 2, Pydantic v1 → v2, OpenAI SDK v0 → v1).
• ▹Rarely used corners. The model has seen too little real usage to anchor on, so it falls back on the pattern.
• ▹Forks and siblings, where a flag really belongs to a related tool (docker vs podman, npm vs pnpm, kubectl vs helm).

The reframe to keep: the more obvious it looks, the more suspicious I am

Most people relax when a line looks obviously right. With AI code, do the opposite. The kwarg or flag that looks like it couldn't possibly be wrong is the one to check, because that's where pattern-completion and the real API are most likely to disagree.

Bonus: if the model guessed wrong at that spot, a human teammate would probably make the same guess. Once you've confirmed the real name, leave a short comment such as `# subset=, not columns= (pandas is inconsistent here)`. You're writing it for the next person, and for the next AI that reads the file.

Split the question in two

Checking AI code really means answering two separate questions, and one is much harder than the other:

• ▹Does this symbol exist? Is there a function, method, kwarg or flag with this exact name in the version I have installed? This one is cheap: one line, a few seconds.
• ▹Does it do what I think? Behaviour, edge cases, defaults, performance. This one is expensive. You need to read docs, write tests or run real data.

In my experience, models are usually right about how popular, real symbols behave, because things like `df.merge` and `json.loads` are all over public code. Where they slip is on whether a symbol exists at all, out at the edges. So check existence first. It's the cheap check, and in my experience it catches most of the damage. Save your expensive checking for code that has already passed it.

The 10-second check: copy-paste block

Run these in your own environment: your virtualenv, your node_modules, your installed CLI version. The model can't check any of that for you. It only knows what was common in its training data. Your terminal knows which version you actually have.

# Python: does the method exist, and what are its real parameters?
python -c "import pandas as pd; help(pd.DataFrame.drop_duplicates)"
python -c "import inspect, pandas as pd; print(inspect.signature(pd.DataFrame.drop_duplicates))"

# Python: does this attribute exist at all?
python -c "import requests; print(hasattr(requests, 'get'))"

# Node: is this export real?
node -e "console.log(typeof require('lodash').chunk)"

# CLI: is this flag real for THIS tool?
mycli --help | grep -- --flag
terraform apply -help | grep -i dry      # empty output = the flag isn't there

# Version drift: is the API the model knows the one I have?
pip show pandas
npm view axios versions --json | tail -5
npm ls axios

• ▹In the IDE, Cmd/Ctrl-click the symbol to go to its definition. If it doesn't resolve, or you land in a generic `**kwargs` signature, treat that as a red flag.
• ▹In Jupyter, `pd.DataFrame.drop_duplicates?` shows the signature instantly.
• ▹Rule of thumb: only probe symbols you haven't used yourself. Checking `print` every time is a waste of your afternoon.

The quiet trap: some fakes don't crash

A TypeError is the good outcome, because you find out straight away. The bad outcome is a fake that gets quietly accepted:

• ▹Functions that take `**kwargs` and forward or ignore them. Wrappers in your own codebase are frequent offenders.
• ▹Lenient CLI parsers that ignore unknown flags or treat them as positional args.
• ▹Config files (YAML, TOML, JSON) that ignore unknown keys. Your invented `retry_backoff: true` does nothing, and the default behaviour runs instead.

Code that runs without an error can still contain a fake. Two quick tells: a warning in the output (pytest, for example, emits `PytestConfigWarning: Unknown config option` for a made-up key in `pyproject.toml`), or a config key you can't find in the official docs or schema. For config files, grep the docs for the exact key before you trust it.

# Silent-failure example: your own wrapper swallows the fake kwarg
def fetch(url, **kwargs):
    return requests.get(url, timeout=kwargs.get("timeout", 10))

fetch(url, retries=3)   # no error, and also no retries

A tiny daily habit loop

• ▹1. Paste into a scratch space first: a Jupyter cell, a REPL or a throwaway terminal. Don't paste straight into the file you're about to commit.
• ▹2. Probe each unfamiliar symbol with the one-line existence check above.
• ▹3. When a probe fails, don't argue with the model. Paste the real `help()` or `--help` output back and say: "This is the actual signature in my installed version. Rewrite using only these parameters."
• ▹4. Wherever the real API is surprising, leave a comment so the next person (or AI) doesn't make the same mistake.

Step 3 saves the most time. Without the real signature, the model will often replace one plausible fake with another. With it, you usually get a correct version in one round trip, because it's working from your actual API rather than guessing.

Carry this into Day 9

Today's rule: make the runtime tell you, not the model. The model is good at proposing code. Only your environment can confirm that a symbol exists in the version you have installed. Later in the series, an agent will run this same check on its own before it acts: it inspects the signature, runs `--help` and reads the installed version before it calls the tool. You'll know why it does that, because you'll have been doing it by hand.

Extend your knowledge

• ▹Python's `inspect` module docs. Learn `inspect.signature` and `inspect.getsource` so you can see what a function really accepts.
• ▹The `requests` docs on Transport Adapters, plus urllib3's `Retry` class. They show the real way to do something a model will happily invent a shortcut for.
• ▹The changelog or migration guide for a library you use every day, such as pandas 2.0 'What's new' or the Pydantic v2 migration guide. These list exactly the renamed and removed symbols that models still suggest.
• ▹Strict modes, wherever your tools have them: type checkers like mypy or pyright, `--strict` flags, schema validation for config files. They reject unknown names before you even run the code.