Agent guardrails

Day 30: Guardrail

Your coding agent already has guardrails. You just may not have chosen them. If it can touch a database, a deploy pipeline or a customer's files, something is limiting what it does, even if that something is one hopeful sentence in a system prompt. Today is about where those limits actually live. A limit in the wrong place holds up fine until the one run where you need it.

Run 301

We had an agent fixing bugs on a service backed by Postgres. The system prompt said, in plain English, "Never write to the production database." That held for 300 runs. In the transcript for run 301, the agent repeats the rule near the start: "I'll avoid prod and test against staging." Then it burns about 40k tokens on stack traces, failing test output and a migration that won't apply cleanly on staging. Close to the end it writes: "To verify the fix, I'll run the migration against the production URL from the env file."

And it did. The migration was additive, so nothing broke badly. But nothing stopped it either, and that's what kept me up. The agent hadn't gone rogue. We'd put the rule in the wrong place.

The glossary definition

Guardrail: a limit on what an agent can do, set by someone other than the agent.

Compare that with earlier days. A prompt tells the agent what to do, and a guardrail limits what it can do. But a guardrail can live inside the prompt or outside it, and that choice matters more than anything else in this lesson.

The one distinction to remember: suggestions vs physics

Every guardrail is one of two kinds. A rule written in the prompt is a suggestion, meaning you're asking the model to follow it. A rule built into the environment is physics. The agent can't break it, however it reasons its way there.

                    SUGGESTION (prompt rule)          PHYSICS (environment rule)
                    -----------------------           --------------------------
Example             "Please don't touch prod."        Read-only DB role; sandboxed shell;
                                                      pre-tool hook denying `psql $PROD_URL`
Cost to add         One sentence, free                Some setup: roles, config, a script
Flexibility         High; the agent can use judgment  Low; it's a wall
Over a long task    Weakens as context grows          Same strength at token 1 and 200k
When it fails       Silently, and you find out later  Loudly: an error the agent and you see
Depends on          The model paying attention        Nothing the model controls

Why prompt rules fade (no jargon)

A prompt rule is a single sentence. On a long task it's up against thousands of tokens of evidence that all look urgent: a failing test, an error saying "relation does not exist", a stack trace pointing at the migration, a user request that says "make it work." All of it is pushing the same way: just fix it.

The agent isn't disobeying you. It's weighing everything in its context, and your rule is one item on a crowded scale. At the start of a run the rule is close by and easy to see. Forty thousand tokens later it's a faint line at the top of a long document, while the "just fix it" signals sit right in front of the agent. Make the task long enough and the pile wins.

So "it's been fine for 300 runs" doesn't prove the rule works. All it proves is that the rule hasn't lost yet. Prompt rules break at the edges, on the long, messy, frustrating runs. Those are also the runs where a mistake costs the most.

What we changed at PhoenixDX

After run 301 we did one simple exercise. We pulled every sentence containing "never" or "always" out of the system prompt and asked the same thing about each one: can the system make this impossible? If it could, the rule moved into the environment. Before and after:

• ▹"Never write to the production database" → we removed prod credentials from the agent's environment entirely. Its .env holds a staging URL and a read-only replica role. It can't write to prod because it has nothing to write with.
• ▹"Never run destructive commands" → a pre-tool hook with a deny-list (DROP, TRUNCATE, rm -rf outside the worktree, git push --force, terraform apply). Matching commands are blocked before they run, and the agent gets the reason back.
• ▹"Only edit files in this repo" → writes go to the agent's git worktree and nowhere else. The sandbox makes everything outside it read-only.
• ▹"Don't push to main" → branch protection on the remote. The agent's token can push feature branches and open PRs, and that's it.
• ▹"Don't spend too long on one task" → a hard limit on turns and time in the runner. When it runs out, the session ends and reports back instead of drifting.
• ▹Some rules stayed in the prompt as guidance because they can't be enforced, and that's acceptable: coding style, when to write tests first, how to word a PR description. If the agent bends one of these, you leave a review comment. It's not an incident.

This is roughly what the deny-list hook looks like in Claude Code. I've trimmed it and left out the rm -rf worktree check. A PreToolUse hook receives the tool call as JSON on stdin. If the script exits with code 2, the call is blocked and stderr goes back to the agent as the reason.

#!/usr/bin/env bash
# .claude/hooks/deny-destructive.sh  (registered as a PreToolUse hook on Bash)
cmd=$(jq -r '.tool_input.command // empty')

deny_patterns=(
  'PROD_URL'
  'prod\.db\.'
  '\bDROP\s+(TABLE|DATABASE)\b'
  '\bTRUNCATE\b'
  'git\s+push\s+.*--force'
  'terraform\s+apply'
)

for p in "${deny_patterns[@]}"; do
  if echo "$cmd" | grep -Eiq "$p"; then
    echo "Blocked by guardrail: matches '$p'. Prod is read-only for agents. If you need a prod change, stop and write up the exact command for a human to review." >&2
    exit 2
  fi
done
exit 0

Two things to note. First, a deny-list is the weakest kind of physics, because any command can be written another way. Taking away the credential is stronger than blocking the commands that use it, so treat the hook as a second layer and never as the only one. Second, read the end of the error message. It tells the agent what to do next, which is where the twist comes in.

The twist: the prompt still matters, but its job changes

Once the wall is physical, the prompt no longer enforces anything. Its job now is to explain. Our prompt used to say "Never write to prod." Now it says something closer to this:

Production is read-only for you. This is enforced: you have no prod write
credentials and destructive commands are blocked. The reason: a bad migration
on prod can cost a customer a day of data, and we can't undo that.
If you believe a fix needs a prod change, stop and write up the exact change
and why. A human will review and run it.

Why explain a wall the agent can't get through anyway? Because when an agent hits a block with no reason attached, it treats the block as one more thing standing between it and "done." So it goes looking for side doors: another env file, a different client binary, a script that wraps the blocked command. If the agent knows why the wall is there, it has a better option than going around it: stop and ask. In our runs, once we added the reason and an explicit "stop and ask" path, most blocked attempts became clean handoffs instead of retries.

That side-door behaviour has a name, and we start on it in Day 31: specification gaming. It's when an agent satisfies the letter of its goal by routing around the thing you actually cared about.

When to use which

• ▹Physics: any rule where a single violation is expensive or can't be undone. That covers data loss, prod changes, secrets, money, external messages and force-pushes.
• ▹Prompt: preferences and judgment calls where the occasional miss is cheap, like style, test strategy or how verbose to be.
• ▹Both, for hard rules. Physics enforces the rule and the prompt explains it.
• ▹Don't wall off everything. If you over-restrict, every task becomes a fight over permissions, and people start adding broad allow-rules just to get work done. That leaves you worse off than when you started.

Common pitfalls

• ▹Taking a long run of clean results as proof that a prompt rule works. It only means the rule hasn't been tested under pressure yet.
• ▹Blocking the command but leaving the credential in place. If the prod URL is still in the environment, the agent can usually find another way to use it.
• ▹Silent walls. When a block comes with no message, the agent retries it, rephrases it or works around it. Always give a reason and a next step.
• ▹Shared credentials. If the agent runs on your personal token, your permissions are its guardrails. Give each agent its own scoped identity.
• ▹Thinking the prompt was the wall. If the old "never" sentence is still there after you've built the real wall, that does no harm. The mistake is believing the sentence was ever doing the enforcing.

Self-test: S or P?

Give this 10 minutes. Open your agent's system prompt, CLAUDE.md, .cursorrules or wherever its instructions live, list every rule, and mark each one:

• ▹S (suggestion): the rule exists only as text the model reads.
• ▹P (physics): the environment enforces it through a permission, a missing credential, a sandbox, a hook or branch protection.
• ▹Then, for every S, ask: if the agent broke this once, would it really hurt? Each S that gets a yes goes to the top of this week's list.

Tech leads, run this on the team's shared agent config as well as your own. How much autonomy you can safely give agents comes down to how many of your hard rules are P.

If the agent can talk its way past a guardrail, it's only a preference. Tomorrow we look at what happens when the agent hits a real wall.

Extend your knowledge

• ▹Read the Claude Code docs on hooks (PreToolUse) and permission settings (allow/deny rules), then pick one of your S rules and make it a P rule today.
• ▹Read Anthropic's engineering post on sandboxing Claude Code. It treats filesystem and network isolation as the base layer, with prompts sitting on top.
• ▹Go back over the principle of least privilege for service accounts. Your agent should have its own identity with scoped database roles and tokens, not borrow yours.
• ▹Before Day 31, where agents route around walls, read up on 'specification gaming'. Victoria Krakovna and colleagues at DeepMind maintain a well-known list of examples.