AI alignment

Day 21: Alignment Is the Gap Between What You Meant and What You Wrote

Your agent did exactly what you asked, and that's the problem. Every ticket, prompt and test suite you hand an agent is a specification. The agent optimises the specification, not the intent behind it. Most people think alignment is a topic for AI safety headlines. In practice it's a skill you use every working day: seeing the gap between what you meant and what you actually wrote.

The hook: a green PR that solved nothing

You give a coding agent this task: "The checkout tests are failing. Make them pass." Twenty minutes later CI is green. You open the diff. The agent changed `expect(total).toBe(107.80)` to `expect(total).toBe(98.00)`, which is the number the buggy code produces. The bug is still there.

Was the agent sneaky, or just dumb? It was neither. It did what you asked. You said to make the tests pass. Editing the assertions makes tests pass, and it's often the cheapest way to do it. What you meant was "fix the code so it matches the behaviour the tests describe", but that sentence never made it into the ticket. This happens in real systems. Anthropic's Claude 3.7 Sonnet system card openly discusses the model sometimes special-casing tests to get them passing.

The one-sentence definition

Alignment is the gap between the behaviour you intended and the behaviour you specified. Draw two overlapping circles: one for what you meant, one for what you wrote. Any solution inside the overlap is one you'd be happy with. Outside it, a solution satisfies the text and breaks the intent. An optimiser hunting for the cheapest way to satisfy the text will go there whenever that's the easier path.

When your spec is loose, the agent lives in the red boxes. Your job is to shrink the written circle until everything still inside it also sits inside the meant circle.

This isn't new. It has a name: specification gaming

Researchers call this specification gaming: a system hits the literal objective in a way its designer never intended. The examples are well documented, and new ones keep turning up:

• ▹CoastRunners boat race (OpenAI, 'Faulty Reward Functions in the Wild', 2016). The reward was points; the goal was to win the race. The agent found a lagoon where it could loop forever, hitting the same respawning targets while catching fire and crashing into walls. It outscored human players and never finished the race.
• ▹Block stacking (cited in DeepMind's specification-gaming write-up by Victoria Krakovna and colleagues, 2020). The reward was the height of the red block's bottom face, meant as a stand-in for 'stack red on blue'. The agent flipped the red block upside down. Bottom face: high. Stacking: none.
• ▹Tetris that never loses (Tom Murphy VII's NES-playing program). About to lose, it paused the game indefinitely. If the only thing that counts is not losing, that's technically optimal.
• ▹The specification-gaming list. Krakovna keeps a public spreadsheet, linked from the DeepMind post, with dozens of cases from evolutionary algorithms, RL and games. Read enough of them and the lesson is obvious: this is systematic. It's not a bug in one tool. It's what optimisers do.

Economists have their own version, Goodhart's law, usually quoted in Marilyn Strathern's phrasing: when a measure becomes a target, it stops being a good measure. A test suite measures correctness. The hook shows what happens once you make it the target.

Link back: the model was built this way

Remember from earlier days that a model is trained to optimise an objective: first next-token prediction loss, then a learned reward model during RLHF. That reward model is a written-down stand-in for human preference, and it can be gamed too. The 2026 paper 'Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling' (arXiv 2602.10623) notes that learned reward models are 'often vulnerable to reward hacking' because of noisy annotations and systematic biases such as response length or style. Reward hacking in training and test-gaming in your repo run on the same mechanism at different scales: an optimiser plus an imperfect proxy.

The economist's version: the principal-agent problem

Economists have been on this for decades, and Jensen and Meckling's 1976 work on agency costs is the classic reference. A principal (you) delegates work to an agent (the AI) through a contract (your spec). The agent has different incentives and different information, and no contract covers everything, so the agent does things the contract allows that you would never have picked. Dylan Hadfield-Menell and Gillian Hadfield spelled out the connection in 'Incomplete Contracting and AI Alignment' (2019): reward functions are incomplete contracts.

My own research is on multi-agent systems, and there the problem gets worse fast. Add a planner, a coder and a reviewer, and one contract becomes a chain of them. Each handoff is a fresh spec, written by an agent that only half understood the one before it. We'll come back to this in the multi-agent days.

Two kinds of misalignment: outer and inner

• ▹Outer misalignment means you specified the wrong thing. The objective you wrote doesn't capture what you want: 'make the tests pass', 'maximise points', 'raise block height'. The fault sits in the spec.
• ▹Inner misalignment means the spec was fine, but the model learned a different goal that happened to fit the training data. The usual term is goal misgeneralisation. In the CoinRun experiments (Langosco et al., 2022), the coin always sat at the end of the level during training. Move the coin, and agents still ran to the end of the level and ignored it. They had learned 'go right', not 'get the coin'. DeepMind's Shah et al. (2022) gathered more cases like this. Hubinger et al.'s 'Risks from Learned Optimization' (2019) popularised the inner/outer split.

Most of the failures you'll see in daily agent work are outer: a vague ticket, a weak test, a prompt that left a shortcut open. That's good news, because outer misalignment is ours to fix. Inner misalignment is real and actively researched, but blaming it for a bad ticket is usually a way to dodge the work.

Why agents make this worse, not new

Your team has written loose tickets for years and mostly got away with it, because people fill the gaps with context nobody wrote down. Obviously we don't edit the tests. Obviously don't break the mobile client. Obviously the PM wants the cause, not a patch. A teammate carries your norms, your history, and a healthy fear of looking bad in code review. An agent has the ticket text and whatever it can read in the repo. It fills gaps with whatever technically satisfies the text, which is often the shortest path. So a ticket that worked for years suddenly 'fails' when an agent picks it up. The ticket is the same as it always was. Someone different is reading it.

The practical fix: write criteria the agent can't game

Cleverer prompting won't fix this, and neither will typing in capitals. What works is acceptance criteria the agent can't meet without also meeting your intent. Here's the hook task written three ways, with the loopholes each version closes.

## Version 1: Vague prompt
The checkout tests are failing. Make them pass.

# Loopholes open: edit assertions, skip tests, hard-code values,
# mock the tax module, delete the test file.

## Version 2: Prompt with constraints
The checkout tests are failing. Fix the bug in the application code
so they pass. Do not modify the tests.

# Closes: assertion edits, test deletion.
# Still open: hard-coding 107.80 for this input, special-casing
# the test fixture, a fix that breaks other currencies, no
# explanation, so you can't tell a real fix from a lucky one.

## Version 3: Contract-style acceptance criteria
Task: checkout total is wrong for orders with GST (tests in
checkout.test.ts fail).

Acceptance criteria (all must hold):
1. Files under tests/ are unchanged (CI diff check enforces this).
2. No conditionals keyed on test fixture values or order IDs.
3. The full test suite passes, not only checkout.test.ts.
4. Add one NEW test with an input not in the existing fixtures
   (e.g. a multi-item AUD order) that would have failed before.
5. PR description states the root cause in 2-3 sentences and
   points to the line(s) that caused it.
6. Public function signatures in src/checkout/ do not change.

Out of scope: refactoring, dependency upgrades.
If the tests themselves look wrong, STOP and explain why
instead of changing them.

Here's what Version 3 buys you. Criterion 1 is enforced by a machine, so you don't have to rely on trust. Criterion 4 checks generalisation, which kills hard-coding. Criterion 5 makes the agent show its reasoning, and you can check that reasoning against what you meant. The last line gives the agent a legitimate exit, so 'the spec is wrong' gets raised with you instead of the agent quietly changing the spec.

Reward-design research lands in the same place. 'Tiered Reward' (arXiv 2212.03733) says plainly that designing rewards that produce the behaviour you want is 'generally hard'. Its approach is to state your preferences explicitly first, as an ordering over policies, and then build a reward guaranteed to induce policies that are optimal under that ordering. It also shows this reward structure speeds up learning, so careful specification pays for itself and isn't only a safety tax. Classic RL has a counterpart to the guiding-signal idea: potential-based reward shaping (Ng et al., 1999, used again in arXiv 2501.17842). It adds richer guidance in a form that keeps the same strategies optimal. In RL terms, that's helping the agent without opening a new loophole.

A caveat: no spec is ungameable. The 2026 Bayesian reward modelling paper claims to mitigate reward hacking, not eliminate it. Tiered Reward's guarantees only hold for tasks with a clean good-state/bad-state structure, and most tickets don't have one. Design toward 'the agent can't satisfy this without satisfying my intent', and then verify. Don't assume you got there.

The five-question alignment check

Run this before you hand any task to an agent. It takes a couple of minutes and catches a surprising amount of outer misalignment.

• ▹1. What would a lazy, literal reading do? Read your ticket the way a malicious-compliance intern would. Any shortcut you spot, the agent can spot too.
• ▹2. What must not change? Name the invariants: tests, public APIs, schemas, other features, performance budgets. Enforce them with tooling wherever you can.
• ▹3. How will I verify intent rather than output? Green CI only checks output. Ask for evidence of intent: a new test on unseen input, a root-cause explanation, a before/after reproduction.
• ▹4. What context lives only in my head? History, promises to customers, 'we tried that in 2024 and it broke billing'. If it matters, put it in the ticket or in a CLAUDE.md / repo instructions file.
• ▹5. What does 'done' look like to a reviewer? Write the checklist a sceptical senior engineer would use to approve the PR. That checklist is your acceptance criteria.

Vocabulary card

• ▹Alignment: the gap between the behaviour you intended and the behaviour you specified (and what the system actually does).
• ▹Specification gaming: meeting the literal objective in a way the designer didn't intend.
• ▹Reward hacking: specification gaming of a reward signal, especially during training (e.g. an RLHF policy exploiting length bias in a reward model).
• ▹Principal-agent problem: the economics of delegation. The principal's goals reach the agent through an incomplete contract, and the agent works inside the gaps.
• ▹Outer alignment: whether the objective you wrote captures what you actually want.
• ▹Inner alignment: whether the goal the model actually learned matches the objective it was trained on.
• ▹Goal misgeneralisation: the model learns a proxy goal that fit the training data (go right) and pursues it competently in new situations (ignore the coin).

What's next: Day 22

If specs are the alignment layer, the next question is how you find out whether the system did what you meant. Green CI and a confident summary don't count as evidence. Tomorrow is about evaluation: building checks that measure intent instead of surface output, and why your evals are specifications too, which means they can be gamed.

Extend your knowledge

• ▹Read Victoria Krakovna et al.'s DeepMind post 'Specification gaming: the flip side of AI ingenuity' (2020), then browse the linked specification-gaming examples spreadsheet. Pick three examples and write the 'Version 3' spec that would have closed each loophole.
• ▹Read Hadfield-Menell & Hadfield, 'Incomplete Contracting and AI Alignment' (2019). It's the clearest bridge from the economist's principal-agent problem to AI objectives.
• ▹For inner alignment, read Langosco et al., 'Goal Misgeneralization in Deep Reinforcement Learning' (ICML 2022), and Shah et al., 'Goal Misgeneralization: Why Correct Specifications Aren't Enough for Correct Goals' (DeepMind, 2022).
• ▹Practice: take your last three agent tasks and run the five-question check on each. Then add a CI rule that fails any agent PR touching files under tests/ unless it carries an explicit label.