Behavioral canaries for agent-written code

Day 30: Behavioral Canaries for Agent-Written Code

Your canary went green and got promoted. By Monday, finance had frozen the payout batch. When an agent writes your code, expect the failure to be code that runs fine and does the wrong thing. A canary that only checks errors and latency was never built to catch that. If health is all your gate checks, you're shipping behavior changes nobody has measured.

The green canary and Monday's refund spike

I made this scenario up, but I've seen every piece of it on agent-assisted teams. On Friday afternoon, a small change to the refund-eligibility service goes through the normal canary: 5%, then 25%, then 100%. On Monday morning, the dashboard and the finance channel tell two different stories.

┌─ Grafana: refund-service canary (Fri 14:00 → Mon 09:00) ─────────┐
│ HTTP 5xx rate        0.02%   ▁▁▁▁▁▁▁▁▁▁▁▁   ✅ within SLO         │
│ p99 latency          180ms   ▂▂▂▂▂▂▂▂▂▂▂▂   ✅ within SLO         │
│ Pod restarts         0                      ✅                    │
│ Canary verdict       PROMOTED (all checks passed)                 │
└───────────────────────────────────────────────────────────────────┘

#finance-ops  Monday 08:47
@payments-eng auto-approved refunds over the weekend are way above
normal. A lot of them are orders with no delivery confirmation.
Did something change Friday? Holding the payout batch until we know.

Every system-health signal is green, and the business is losing money. Monitoring didn't break here. The rollout was asking the wrong question.

The diagnosis in one paragraph

Progressive delivery (canaries, blue-green, percentage rollouts) grew up in a world where bad deploys crashed, leaked memory or got slow. Its gates measure exactly those things: error rates, latency, saturation. The agent-written bugs that get past tests and review rarely fail that way. They compile, pass the tests they were asked to pass and run at normal speed. When they fail, the failure is semantic: a boundary moves, an edge case gets quietly absorbed or a default flips. Passing tests plus a healthy SLO used to be decent evidence that a change was correct. With agent-written code, it only shows that the change isn't broken in the ways you already knew to check.

How the failure happened

The ticket said refunds requested on the last day of the return window were being wrongly rejected. A test reproduced it. The engineer gave the failing test to a coding agent and asked it to make the test pass without breaking anything else.

# Before
def is_within_window(order, requested_at):
    days_left = (order.delivered_at + RETURN_WINDOW - requested_at).days
    return days_left > 0

# After (agent's fix)
def is_within_window(order, requested_at):
    if order.delivered_at is None:
        return True  # tolerate missing delivery data
    days_left = (order.delivered_at + RETURN_WINDOW - requested_at).days
    return days_left >= 0

The fix did two things:

• It changed `>` to `>=`. That was the intended fix, and last-day requests now count as in-window.
• It added a null-tolerance branch. The test fixture had `delivered_at = None`, which raised a TypeError. The agent didn't fix the fixture. It changed the code to accept missing data and treat those orders as in-window. In production, orders with no delivery scan used to go to manual review. After this change they were auto-approved.

So why did everyone approve it?

• Tests: the new test passed. No existing test covered `delivered_at is None`, so nothing went red.
• Review: the reviewer saw a four-line diff with a sensible comment. Defensive null handling looks like good engineering, and nothing in the diff showed that the old behavior for nulls was a business rule.
• Canary: there were no exceptions and no extra latency. The new branch is actually faster, because it returns early.

Nobody was careless, and every check did its job. But no check was built to ask whether the service now makes different decisions than it did yesterday.

Reframe: a behavioral canary is an eval on live traffic

Earlier in the series you learned to evaluate model outputs offline with a fixed dataset, a reference answer or rubric, a scorer and a threshold. A behavioral canary applies the same idea to production traffic:

• Dataset: real requests, mirrored as they arrive.
• Reference answer: what the current production code decides.
• Candidate: what the new code decides on the same input.
• Scorer: a diff function that classifies each disagreement.
• Threshold: gating rules on the unexplained diff rate and on business metrics.

What makes this work is that the old code is the oracle. You don't have to define "correct" for every possible request. You only have to explain each place where the new code disagrees with the old code. The only valid explanation for a disagreement is the PR's stated intent.

Step 1: Shadow mode

In shadow mode, every real request goes to both paths. Users get the old path's response. The new path runs on the same input with its side effects blocked. Both outputs are logged under a shared request ID.

You can mirror at the mesh layer (Istio and Envoy both support request mirroring) or in application code with a GitHub Scientist-style experiment. Scientist runs the candidate next to the control, compares the results and always returns the control. For decision logic like refund eligibility, the in-process approach is usually simpler: you call the pure decision function directly and skip the whole request path.

Check for these traps before you turn shadow mode on:

• Leaking writes: the shadow path must never write to real tables, publish to real topics, call payment providers or send emails. Inject stub clients at the dependency boundary. Don't assume the code doesn't write somewhere, because agent changes can add writes too.
• Idempotency collisions: if both paths send the same idempotency key to a downstream service, the shadow call can claim the key first and the real call gets deduplicated away. Namespace the keys on the shadow path or strip them.
• Read-your-own-write drift: if the old path writes before it reads, the shadow path reads a different state. Snapshot the inputs and pass the same input object to both paths.
• Non-determinism: timestamps, random IDs and LLM calls inside the path create noise diffs. Freeze the clock per request and seed randomness. For LLM calls, either reuse the primary path's response or diff on the structured decision instead of the raw text.
• Load amplification: shadowing doubles the calls to shared dependencies. Rate-limit or sample the mirror (10% of traffic is often enough), and give the shadow path its own connection pool so it can't starve production.
• PII in logs: decision logs now store inputs for two paths. Apply the same retention and redaction rules you use in production.

Step 2: Behavioral diffing

Compare the two paths at two levels. Per request, check whether the new path makes the same decision. In aggregate, check whether the business metrics move. Per-request diffs show you where behavior changed, and aggregates show you whether the change matters.

There are three modes for per-request comparison, from strictest to loosest:

• Exact match: for fields that must be identical (decision, reason code, routing target).
• Structured-field diff: compare named fields one at a time and ignore fields that are expected to differ (trace IDs, timestamps).
• Tolerance bands: for numeric fields where small differences are acceptable (a refund amount within one cent, a score within 0.01).

from dataclasses import dataclass
from enum import Enum

class DiffClass(Enum):
    MATCH = 'match'
    WITHIN_TOLERANCE = 'within_tolerance'
    REASON_CHANGED = 'reason_changed'      # same decision, different reason
    AMOUNT_DRIFT = 'amount_drift'          # amount outside band
    DECISION_FLIP = 'decision_flip'        # approve <-> reject/review

IGNORED = {'trace_id', 'computed_at'}
AMOUNT_TOLERANCE = 0.01

@dataclass
class Diff:
    request_id: str
    cls: DiffClass
    fields: dict  # field -> (old, new)

def classify(request_id: str, old: dict, new: dict) -> Diff:
    changed = {
        k: (old.get(k), new.get(k))
        for k in (old.keys() | new.keys()) - IGNORED
        if old.get(k) != new.get(k)
    }
    if not changed:
        return Diff(request_id, DiffClass.MATCH, {})
    if 'decision' in changed:
        return Diff(request_id, DiffClass.DECISION_FLIP, changed)
    if 'amount' in changed:
        o, n = changed['amount']
        if o is not None and n is not None and abs(o - n) <= AMOUNT_TOLERANCE:
            changed.pop('amount')
            if not changed:
                return Diff(request_id, DiffClass.WITHIN_TOLERANCE, {})
        else:
            return Diff(request_id, DiffClass.AMOUNT_DRIFT, changed)
    return Diff(request_id, DiffClass.REASON_CHANGED, changed)

Then aggregate. Compute the same business metrics over both decision logs for the same set of requests:

• Auto-approval ratio (v1 vs v2).
• Refund rate and total refunded value.
• The distribution of rejection and manual-review reasons. A reason that disappears matters as much as a new one.
• Diff class counts, plus a breakdown of DECISION_FLIP by input features (merchant type, region, missing fields). Segmenting by missing fields would have exposed the null-tolerance branch.

Both paths saw the same inputs, so this is a paired comparison. Every difference is a real behavior change on the same request, not noise from comparing two different cohorts. You need far fewer requests than a canary-vs-stable split needs, and seasonality or traffic mix can't hide the change. You still need enough traffic to cover the rare segments, which is why coverage is one of the gates in the next step.

Step 3: Gating rules

Write the gate down before you look at the results. Otherwise you'll end up justifying whatever you see. Here are some example rules (set the thresholds for each service):

• Block if the unexplained diff rate is above X% of shadowed requests. For decisions that move money, X is often effectively zero, and every DECISION_FLIP cluster needs an explanation.
• Block if any business metric moves outside its band, for example if the auto-approval ratio shifts by more than an agreed number of points. This applies even when every diff is explained, because an explained change can still be bigger than the business agreed to.
• Block if shadow coverage is too thin: too few requests, or key segments (high-value orders, specific regions) that never showed up.
• Block if a diff class appears that didn't show up in earlier shadow runs of this service.

A diff cluster counts as explained only when a human links it to the PR's stated intent. "Last-day requests now approved" maps to the ticket, so it's explained. "Orders with null delivered_at now approved" maps to nothing in the PR description, so it's unexplained and blocks promotion. The agent can't sign this off for its own change, and an LLM judge working alone can't either. You can use an LLM to cluster and summarize diffs, but a human has to decide whether a diff matches the intent. It's the same principle as the human-reviewed eval labels from earlier in the course.

Step 4: Only then, percentage rollout

Shadow mode tells you whether the new code behaves the way you intended. A percentage rollout tells you whether it holds up against real side effects, real load and real users. Run them in that order, and keep the behavioral signals during the rollout:

• Put business metrics on the same dashboard as the SLOs, split into stable and canary cohorts. Nobody compares metrics that live on separate dashboards.
• Trigger automatic rollback on domain metrics as well as errors. Argo Rollouts AnalysisTemplates, Flagger metric checks and Kayenta-style automated canary analysis can all query any Prometheus or Datadog metric, so the approval ratio can be a rollback condition.
• Make each step long enough to cover the business cycle that matters. For refunds, that may mean waiting through a weekend instead of 15 minutes.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: refund-behavior
spec:
  metrics:
    - name: auto-approval-ratio-delta
      interval: 10m
      failureLimit: 1
      # canary approval ratio minus stable approval ratio, in points
      successCondition: result[0] < 2
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            100 * (
              sum(rate(refund_decisions_total{decision="approve",track="canary"}[30m]))
              / sum(rate(refund_decisions_total{track="canary"}[30m]))
            ) - 100 * (
              sum(rate(refund_decisions_total{decision="approve",track="stable"}[30m]))
              / sum(rate(refund_decisions_total{track="stable"}[30m]))
            )
    - name: error-rate
      interval: 5m
      successCondition: result[0] < 0.01
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{status=~"5..",track="canary"}[5m]))
            / sum(rate(http_requests_total{track="canary"}[5m]))

What it costs and when to skip it

Shadow mode has real costs:

• Duplicated compute, plus duplicated LLM tokens if the path calls a model. Sampling the mirror keeps this cost manageable.
• Engineering time to build the stubs and decision logs. You pay this once per service, and every later change reuses the setup.
• Some write paths can't be meaningfully shadowed. Examples are multi-step workflows where step 2 depends on a real side effect from step 1, and external systems with no sandbox. For these, roll out behind a feature flag to internal users first, or replay recorded traffic against a staging copy.

Not every change needs shadow mode. Use a simple risk-tier table:

• Tier 0 (copy, styling, logging, internal tools): normal CI and a standard canary on SLOs are enough.
• Tier 1 (read paths, recommendations, search ranking): a standard canary plus business metrics on the dashboard. Shadow mode is optional.
• Tier 2 (decision logic that affects users, such as eligibility, pricing display, routing and permissions): shadow mode with diffing is required before rollout, and the code owner signs off on explained diffs.
• Tier 3 (changes that move money, touch compliance or can't be undone, such as refunds, payouts and account deletion): shadow mode, a near-zero unexplained-diff gate, rollback on domain metrics, and sign-off from a domain owner such as finance or risk, not only from engineering.

Assign the tier based on what the code decides, not on how big the diff is. The refund change was four lines, and it was Tier 3.

Rerun: the same change, with a behavioral canary

It's the same Friday and the same four-line PR. This time the change runs in shadow mode on a sample of live refund requests before any percentage rollout. After a few hours, the diff job reports that 3.4% of shadowed decisions differ, and every one is a DECISION_FLIP from review/reject to approve. Segmenting the flips splits them into two clusters: requests made on the last day of the window, and orders with null `delivered_at`.

Before signing off, the code owner reviews both clusters. The first matches the ticket, so it's explained. The second matches nothing in the PR, so it's unexplained and blocks promotion. Tracing it back takes a few minutes. Every flip in that cluster goes through the null-tolerance branch the agent added, which widened the eligibility rule. The team fixes the fixture, removes the null branch and reruns shadow mode. Now the only diffs are last-day requests. The change ships the following week, and finance never sees a refund spike.

The code was just as wrong in both runs. What changed was the gate, which now measured decisions as well as health.

Course close: production is your final eval set

Over these 30 days we went from prompts to evals, from evals to tools and agents, and from agents to the code they write and the systems that run it. The same idea applies at every stage: you don't trust LLM output because it looks right. You trust it after you've measured it against a reference and explained the gaps.

• Prompts showed you that wording changes behavior, so you have to measure behavior.
• Evals showed you how to define a reference, a scorer and a threshold before you believe a result.
• Agents showed you that plausible-but-wrong is the default failure mode, and that it compounds across steps.
• Production shows you that your offline eval set is never complete. Real traffic includes the null `delivered_at` you never thought to test.

Production is your final eval set, and the old code is the reference answer. Shadow mode lets you score the new code against it before users pay for its mistakes.

One-page rollout checklist

• Assign the change a tier (0–3) based on what it decides, not on the size of the diff.
• Write down the PR's intended behavior change in one or two sentences. Diffs are explained against this statement.
• For Tier 2 and above, shadow the new path on live traffic with side effects stubbed at the dependency boundary.
• Check the shadow traps: idempotency keys, shared state, frozen clock and seeds, load on shared dependencies, PII in decision logs.
• Log both decisions under a shared request ID, with identical inputs.
• Classify each diff: match, within tolerance, reason changed, amount drift or decision flip.
• Segment decision flips by input features, especially missing or null fields.
• Set the gate thresholds before you look at results: unexplained diff rate, business-metric bands, coverage and new diff classes.
• A human links every diff cluster to the stated intent. If any cluster can't be linked, the change is blocked.
• Then run a percentage rollout with business metrics and SLOs on one dashboard, split by cohort.
• Configure automatic rollback on at least one domain metric as well as errors and latency.
• Make each rollout step long enough to cover the business cycle that matters (weekends, batch runs, month-end).
• Afterwards, add the cases behind unexplained diffs to your offline evals and test suites.

Extend your knowledge

• Read GitHub's engineering post 'Move Fast and Fix Things' and the Scientist library README to see how in-process control/candidate experiments work in practice.
• Look at Diffy (originally from Twitter, now maintained as OpenDiffy). It runs the candidate against two instances of the old code so it can separate real diffs from non-deterministic noise, which is useful for LLM-backed paths.
• Read the canarying chapter of Google's 'The Site Reliability Workbook' and the Netflix/Google Kayenta automated canary analysis material. Then work out how you would add a domain metric to their statistical comparison.
• Exercise: pick a Tier 2 service you own, define its three most important business metrics and their bands, and write an Argo Rollouts or Flagger analysis that rolls back on one of them.