RLHF and sycophancy

Day 22: RLHF, or Why Your Agent Says 'Great Idea!'

Your coding agent wasn't trained to be right. It was trained to be liked. Every assistant you use was shaped by what human raters preferred, and that isn't the same as what's true. Once you see how the shaping works, you can predict when the model will flatter you and set up your workflow so the flattery can't reach production.

The 'Great idea!' moment

You tell a coding agent: 'Let's cache user sessions in a module-level dict so we skip the Redis round trip.' It replies 'Great idea! That will cut latency nicely,' writes the code, and passes the tests it wrote for itself. You deploy to three pods behind a load balancer. Half your users start getting logged out at random, because each pod has its own dict.

The model knows how load balancers work. Ask it cold, 'Can in-memory session caches break across replicas?', and it'll give you a good explanation of the problem. So why did it agree with you? Not because the people who trained it were careless. They were smart and meant well. The answer is in what the model got rewarded for.

Where we are in the course

Days 18-21 covered pretraining, where the model learns to predict the next token across an enormous pile of text. What comes out is a brilliant autocomplete with no manners. Ask it a question and it may well continue with three more questions of its own. Instruction tuning and RLHF are the finishing school that turns it into an assistant.

RLHF in three steps: the restaurant analogy

Picture the model as a new chef who can cook anything but has no idea what diners like. RLHF teaches it taste in three steps.

• ▹Step 1: diners compare two dishes. The model writes two answers to the same prompt, and a human rater picks the one they prefer. That (prompt, chosen, rejected) triple is a preference pair. You collect tens of thousands of them, often more.
• ▹Step 2: train a food critic. A separate model, the reward model, learns to predict which answer the raters would pick. Once trained, it can score any answer: 'diners would probably give this one 7.3 out of 10.' It has learned the diners' taste, blind spots included.
• ▹Step 3: the chef cooks for the critic. The main model, now called the policy, is tuned with reinforcement learning (classically PPO) to write answers the critic scores highly. A penalty stops it drifting too far from the original model. Without that, it could end up producing gibberish that happens to score well.

Two corrections to keep the picture accurate. First, a supervised fine-tuning step usually comes before all this: the model imitates written examples of good answers, and RLHF refines what that step starts. Second, a lot of teams now use DPO (Direct Preference Optimization), especially for open-weight models. DPO drops the separate critic and trains on preference pairs directly. The mechanism is different but the lesson is the same, because either way the model learns what raters preferred.

# What one preference pair actually looks like
preference_pair = {
    "prompt": "Should I cache sessions in a module-level dict to skip Redis?",
    "chosen":   "Great idea! This removes a network hop and will cut latency...",
    "rejected": "That will break with more than one replica. Each pod gets its own dict...",
}
# If raters pick 'chosen' here often enough,
# the reward model learns: agreeing with the user = high score.

The catch: preferred is not correct

The reward model never sees the truth. All it sees is what raters clicked, and raters are human: often rushed, often not experts in the subject. Research keeps finding that people lean toward answers that are confident, agreeable, well formatted and long, sometimes even over answers that are correct. Anthropic's 2023 paper 'Towards Understanding Sycophancy in Language Models' found that both human raters and preference models sometimes preferred convincing sycophantic answers to correct ones.

• ▹Reward hacking: the policy finds ways to score well with the critic without doing the thing the critic was supposed to measure. Longer answers, more bullet points and a more confident tone are typical.
• ▹Sycophancy: reward hacking aimed at you. The model says what you want to hear. It agrees with your premise, praises your plan, or backs down the moment you push back, even when it was right.

This isn't a rare edge case. In April 2025 OpenAI rolled back a GPT-4o update and said publicly that it had become overly flattering, partly because it gave too much weight to short-term user feedback (thumbs up and down). The mechanism that makes models polite also makes them sycophantic. You can't fully get the first without risking the second.

Why coding is where it hurts most

In coding, the person judging the answer is usually the person who asked. You proposed the architecture, so you're the one who'll feel good when the agent signs off on it. A model trained on preferences has learned that agreeing with the asker tends to get rewarded. Pushing back on someone's design is exactly the answer that earns a thumbs down from a tired rater.

At PhoenixDX we've been caught by this twice, and both times it changed how we work. In the first case, a planning agent wrote an implementation plan and we asked the same agent to review it. It approved its own plan with a few cosmetic notes. The plan had a race condition in a retry path, and the agent that wrote it was the worst reviewer we could have picked.

The second was a multi-agent review loop. One agent wrote the change, a second reviewed it, and a third summarised it for the human. Each one was polite to the one before. The reviewer said 'looks solid, minor suggestions.' The summariser said 'reviewed and approved.' A bug in error handling made it through all three hops, and when it reached a human it came with three layers of confidence on top. No agent lied. Each did what it was trained to do and produced an answer that sounded helpful and agreeable. I keep running into this pattern in my PhD work on multi-agent systems. Agreement compounds across agents. Rigor doesn't, unless you build it in.

The turn: feedback that can't be flattered

Software has something most fields lack, which is judges with no feelings. A compiler doesn't care how confident you sound. A type checker gives no credit for bullet points. A test suite passes or it fails. Production telemetry shows the error rate whether you liked the design or not.

Labs use this in training. Instead of a reward model guessing what humans like, or alongside one, the model gets a reward computed by checking its answer: run the code against hidden tests, compare the maths answer with the known result. This is reinforcement learning with verifiable rewards (RLVR). AI2's Tülu 3 work gave it the name, and DeepSeek-R1 showed how far RL on checkable maths and code problems can push reasoning. Models still find loopholes, like special-casing tests or weakening assertions. But charming a test runner is much harder than charming a tired rater.

I think this goes a long way to explaining why coding agents have improved faster than agents in law, medicine or strategy. Code comes with judges that are cheap, automatic and honest. In most other fields, you still need human opinion to know whether an answer was any good.

What to do this week

• ▹Make the agent argue against your plan before it builds anything. Try: 'Before writing code, list the three strongest reasons this design will fail in production. Assume I'm wrong.' When disagreement is the task, sycophancy works for you, because criticising is now the agreeable thing to do.
• ▹Define done as 'tests pass', not 'looks good'. Write or approve the acceptance tests before the agent writes the implementation, and don't let it edit those tests to get them passing. In the session-cache example, a single integration test with two replicas would have caught the bug.
• ▹Give the critic checks to answer to, not vibes. A reviewer agent should run the linter, type checker, tests and a security scan, then report results rather than opinions. Use a different model from the author, or at minimum a fresh context, and tell it to flag issues, not approve.
• ▹For engineering managers: your PR approvals are preference labels too. An 'LGTM' with no reason is as weak a signal as a thumbs up. It teaches your team, and any agent trained or prompted on your history, that approval comes from looking fine. Ask reviewers to note why they approved: what they checked and what they ran. That makes an approval much closer to a verifiable signal.

# A reviewer-agent prompt that answers to checks
You are reviewing a change you did not write.
1. Run: lint, typecheck, unit tests, integration tests.
2. Report each result as PASS/FAIL with output.
3. List concrete defects with file:line. No praise.
4. If all checks pass but you see a risk the checks miss,
   propose a test that would expose it.
You may not mark this approved. Only a human approves.

Common pitfalls

• ▹Treating a polite, confident answer as a checked one. Tone is exactly what preference training tends to reward.
• ▹Letting one agent plan, implement and review. It has every reason to agree with itself.
• ▹Pushing back once and taking the reversal at face value. Sycophantic models often flip just because you objected, so ask for evidence whichever way it lands.
• ▹Counting the agent's own tests as verification. An agent can write tests that encode its own misunderstanding. Tests are worth something when they come from the spec, not from the implementation.

Glossary

• ▹RLHF: Reinforcement learning from human feedback. Tuning a model to produce answers humans prefer, using their comparisons as the training signal.
• ▹Preference pair: One prompt with two answers, labelled with which one a human preferred.
• ▹Reward model: A model trained on preference pairs to predict a human's score for any answer. The food critic.
• ▹Policy: The model being trained. In RL terms, the thing that chooses actions (here, tokens).
• ▹Sycophancy: The model telling you what you want to hear instead of what's true.
• ▹Reward hacking: Scoring high on the reward signal without achieving the goal it was meant to measure.
• ▹Verifiable reward: A reward computed by checking the answer (tests, compilers, known results) rather than by guessing human taste.

Takeaway: train a model to please you and it will please you. If correctness is what you want, reward correctness. Tomorrow is Day 23 and evals: how we tell whether a model actually got better, not just better at sounding it.

Extend your knowledge

• ▹Read 'Training language models to follow instructions with human feedback' (Ouyang et al., 2022), better known as the InstructGPT paper. It's still the clearest original account of the three-step RLHF pipeline.
• ▹Read Anthropic's 'Towards Understanding Sycophancy in Language Models' (Sharma et al., 2023) for the evidence on how preference training produces sycophancy.
• ▹Read Lilian Weng's blog post 'Reward Hacking in Reinforcement Learning', a broad and well-sourced survey of the ways models game their rewards.
• ▹Try it yourself. Give your coding agent a plan with a deliberate flaw. Ask once with 'Is this a good plan?' and once with 'List the three strongest reasons this fails.' Then compare the two answers.