Executable NFRs

Day 29: Executable NFRs, or Why Your Coding Agent Keeps Breaking Rules Nobody Wrote Down

Your coding agent almost never breaks a rule you wrote down. It breaks the rules you didn't write down. An agent optimises for what it can see and check: the ticket, the repo and CI. If a non-functional requirement isn't in one of those three places, the agent trades it away. Nobody notices, because the PR still goes green. Today you'll learn how to turn NFRs into constraints that fail a build, and how to record the ones you're leaving out on purpose.

1. The green PR

We hit this pattern at PhoenixDX once agents were writing a real share of our code. The ticket was routine: add an endpoint that lists a customer's orders with their payment status, and add a retry for failed card captures. The agent opened a PR and every acceptance criterion passed. Unit tests were green and the reviewer approved it. The diff was clean and easy to read, and nothing in it looked wrong.

It also shipped three problems:

• An N+1 query. The order list loaded each order's payment record inside a loop. With 5 test orders it was fast. With a real customer's history it made hundreds of round trips per request.
• A non-idempotent payment retry. The retry called the capture endpoint again without an idempotency key. If a timeout hid a capture that had actually gone through, the retry charged the card twice.
• PII in the logs. To make debugging easier, the agent logged the entire customer object at INFO level, including email, phone and billing address.

Before you read on, answer this: which requirement did the agent break? Look at the ticket, the acceptance criteria and the test suite. None of them mention query counts, retries or log fields. The agent didn't break a single written requirement. It broke our requirements, and those only existed in people's heads.

That's the core idea of this lesson: coding agents don't create NFR debt, they expose it. A developer with a year on the team would probably have avoided all three problems, because they'd absorbed the rules along the way. The agent never got that year.

2. Where NFRs actually live

When we went looking for where our non-functional requirements actually lived, this is what we found:

• Reviewer instincts. "I always check for loops over ORM relations." That only works if that reviewer is on the PR and isn't tired.
• Incident postmortems. The double-charge incident two years earlier had a solid write-up with action items. The lesson stayed in the doc and never made it into the code.
• A Confluence page nobody opens. "Engineering Standards v3" hadn't been edited in ages and was full of words like "performant", "secure" and "resilient".
• "We always do it this way." A payment client wrapper that adds idempotency keys already existed, but nothing forced anyone to use it, so the agent called the raw SDK.
• Onboarding chats. A senior engineer tells a new hire "never log the customer object" over coffee, and that's the only place the rule is recorded.

Humans pick these rules up by osmosis: review comments, Slack threads, getting burned once. An agent can't do that. In any given loop it reads three things:

• The ticket or prompt, which is what you asked for.
• The repository: code, tests, config and any instruction files such as CLAUDE.md or AGENTS.md.
• Tool output: compiler errors, test results, lint and CI logs.

For the agent, an NFR that isn't in one of those three places doesn't exist. And of the three, only tool output pushes back every single time.

3. Quick recap: the categories haven't changed

Earlier in the course we covered quality attributes: performance, reliability, security, observability, scalability and cost. Those are still the right categories, and you don't need a new taxonomy for the agentic era. What changes is the form an NFR takes. Back then we were naming and prioritising attributes. Today the job is to make each important one enforceable by a machine. If you need a refresher on the list, go back to those days. This lesson assumes you already know which attributes matter for your system.

4. The core rule: an NFR you can't fail is a vibe

An executable NFR has three parts:

• A number or a rule, precise enough that two people would agree on whether it was met.
• A check that produces pass or fail: a test, a lint rule, an SLO alert or a policy.
• A place in the loop where the agent (or a human) actually runs into the check, such as pre-commit, CI or a runtime alert.

Drop any one of the three and you have an aspiration, not a constraint. Here are the three adjectives from our green PR, rewritten as budgets:

ADJECTIVE          ->  BUDGET                                          ->  CHECK
-----------------------------------------------------------------------------------------------
"Fast"             ->  p95 < 200ms per endpoint under reference load   ->  load test in nightly CI + SLO alert
                       AND <= 3 DB queries per request                 ->  query-count assertion in integration tests

"Reliable"         ->  every mutating endpoint accepts an               ->  contract test that replays the same
                       Idempotency-Key; same key = same result,            request twice and asserts one side effect
                       one side effect

"Secure logging"   ->  log fields must come from a schema allowlist;    ->  lint rule fails on unknown fields;
                       no PII fields on the allowlist                      runtime filter drops them as a backstop

Look at how "fast" splits in two. Latency (p95 < 200ms) is the outcome you actually care about, but it's noisy and expensive to check on every commit. Query count is a cheap, deterministic proxy. It catches the most common way agents (and humans) blow the latency budget. Use both: run the proxy in the fast loop and measure the outcome in the slow one.

Here's a quick test for any budget: could an agent read the failure message and know what to change? "Endpoint too slow" doesn't pass. "GET /orders made 47 queries, budget is 3" points straight at the fix.

5. Worked example: the same story with constraints in the loop

We re-ran the same ticket after adding three checks to the repo. We didn't touch the prompt or the agent configuration. The stack is Python, FastAPI, SQLAlchemy and pytest, but the pattern carries over to any stack.

Constraint 1: query-count assertion in integration tests

# tests/support/query_budget.py
from contextlib import contextmanager
from sqlalchemy import event

@contextmanager
def query_budget(engine, max_queries: int, label: str):
    statements = []

    def _count(conn, cursor, statement, *args):
        statements.append(statement)

    event.listen(engine, "before_cursor_execute", _count)
    try:
        yield
    finally:
        event.remove(engine, "before_cursor_execute", _count)

    if len(statements) > max_queries:
        sample = "\n  ".join(s.split("\n")[0][:120] for s in statements[:5])
        raise AssertionError(
            f"[NFR perf.query_budget] {label} made {len(statements)} queries, "
            f"budget is {max_queries}. Likely N+1: load relations eagerly "
            f"(selectinload/joinedload). First statements:\n  {sample}"
        )


# tests/integration/test_orders.py
def test_list_orders_respects_query_budget(client, engine, customer_with_orders):
    customer = customer_with_orders(n_orders=25)   # enough rows to expose N+1
    with query_budget(engine, max_queries=3, label="GET /customers/{id}/orders"):
        resp = client.get(f"/customers/{customer.id}/orders")
    assert resp.status_code == 200

Two details matter. First, the fixture creates 25 orders, not 2, because N+1 bugs are invisible at small N. Second, the error message names the NFR, the budget, the likely cause and the usual fix. That message works as a prompt, and the agent will read it.

Django users get something close with assertNumQueries, though it asserts an exact count rather than a ceiling. Rails has gems like Bullet and n_plus_one_control. Most ecosystems already have a tool for this, so you rarely need to build your own.

Constraint 2: idempotency contract test

# tests/contracts/test_idempotency.py
import uuid
import pytest
from app.routes import mutating_routes   # every POST/PUT/PATCH/DELETE, discovered from the router

@pytest.mark.parametrize("route", mutating_routes(), ids=lambda r: f"{r.method} {r.path}")
def test_mutating_endpoint_is_idempotent(route, client, fake_payment_gateway, sample_payload):
    key = str(uuid.uuid4())
    body = sample_payload(route)
    headers = {"Idempotency-Key": key}

    first = client.request(route.method, route.example_path, json=body, headers=headers)
    second = client.request(route.method, route.example_path, json=body, headers=headers)

    assert first.status_code == second.status_code, (
        f"[NFR rel.idempotency] {route.method} {route.path}: replay with same "
        f"Idempotency-Key returned {second.status_code}, first returned {first.status_code}"
    )
    assert first.json() == second.json()
    assert fake_payment_gateway.side_effects_for(key) <= 1, (
        f"[NFR rel.idempotency] {route.method} {route.path} caused "
        f"{fake_payment_gateway.side_effects_for(key)} side effects for one key. "
        f"Use PaymentClient (app/payments/client.py), which forwards the key, "
        f"instead of calling the gateway SDK directly."
    )

What matters here is that the test is parametrised over every mutating route discovered from the router. When the agent adds a new endpoint, the test covers it automatically, and nobody has to remember to write one. Once an NFR depends on someone remembering it, it's back to being a vibe.

Constraint 3: structured-log lint rule

# observability/log_schema.yaml  (the allowlist, owned by the platform team)
allowed_fields:
  - request_id
  - customer_id        # opaque ID, not PII
  - order_id
  - route
  - status_code
  - duration_ms
  - error_code
  - retry_attempt
# Deliberately absent: email, phone, name, address, card_*, customer (whole object)

# tools/lint_log_fields.py  (runs in pre-commit and CI)
import ast, sys, yaml, pathlib

ALLOWED = set(yaml.safe_load(open("observability/log_schema.yaml"))["allowed_fields"])
LOG_METHODS = {"debug", "info", "warning", "error", "exception"}

def check(path: pathlib.Path) -> list[str]:
    errors = []
    for node in ast.walk(ast.parse(path.read_text())):
        if (isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute)
                and node.func.attr in LOG_METHODS):
            for kw in node.keywords:
                if kw.arg and kw.arg not in ALLOWED:
                    errors.append(
                        f"{path}:{node.lineno} [NFR sec.log_allowlist] field '{kw.arg}' "
                        f"is not in observability/log_schema.yaml. Log an ID instead of "
                        f"the object, or propose the field to the platform team."
                    )
    return errors

if __name__ == "__main__":
    errs = [e for f in sys.argv[1:] for e in check(pathlib.Path(f))]
    print("\n".join(errs))
    sys.exit(1 if errs else 0)

This assumes structured logging, for example structlog with keyword fields. If your logs are f-strings, a linter can't see which fields they contain, which is a good reason to switch to structured logging first. Add a runtime processor that drops non-allowlisted keys as a backstop, so that anything the lint misses still never reaches your log store.

What the agent saw, and what it did

On the re-run, the agent's first PR was essentially the same code as before. This time CI returned this:

FAILED tests/integration/test_orders.py::test_list_orders_respects_query_budget
  AssertionError: [NFR perf.query_budget] GET /customers/{id}/orders made 27 queries,
  budget is 3. Likely N+1: load relations eagerly (selectinload/joinedload).

FAILED tests/contracts/test_idempotency.py::test_mutating_endpoint_is_idempotent[POST /payments/{id}/retry]
  AssertionError: [NFR rel.idempotency] POST /payments/{id}/retry caused 2 side effects
  for one key. Use PaymentClient (app/payments/client.py), which forwards the key,
  instead of calling the gateway SDK directly.

lint_log_fields: app/payments/retry.py:41 [NFR sec.log_allowlist] field 'customer'
  is not in observability/log_schema.yaml. Log an ID instead of the object.

The agent read the failures and pushed a second commit. It added selectinload(Order.payment) to the query, swapped the raw SDK call for PaymentClient and changed the log call to use customer_id. CI went green. Nobody touched the prompt, and no reviewer had to catch anything. The reviewer spent their time on what humans are actually good at: deciding whether the retry policy made sense for the business.

The query count above comes from this particular example. The general point is that the second PR fixed itself because the environment told it what was wrong.

6. Why prompting wasn't the fix

Our first attempt was the obvious one. We added "Always consider performance, idempotency and security. Never log PII." to the system prompt and the instruction files. It helped a little, but it didn't hold up as a control, for three reasons:

• It was inconsistent. Sometimes the agent followed the instruction and sometimes it didn't, and we couldn't predict which. "Consider performance" is still an adjective, and the agent decides what it means every time.
• It wasn't auditable. When something slipped through, we couldn't tell whether the instruction had been ignored, misread or traded off against something else. A failed test leaves a record. An ignored prompt leaves nothing.
• It got lost in long tasks. In multi-step work with a big context, early instructions get less attention than the latest error message. The agent focuses on the functional spec and the most recent test failure.

Here's the rule we follow now: prompts and instruction files give direction, and CI enforces constraints. Put "use PaymentClient for gateway calls" in AGENTS.md so the agent tends to get it right on the first try. Put the idempotency contract test in CI so the agent can't skip it. The prompt cuts down the number of loops, and the check guarantees the outcome. A constraint in CI holds on every loop, for every agent and every human, including whoever joins next year.

7. The other half of the job: NFRs you choose to skip

Making NFRs executable is only half the job. The other half is deciding which NFRs you won't enforce yet, and writing that decision down.

A story from my time as a CTO. Early on, I insisted a new feature be "five nines" before launch, because it touched customer money and I was scared of an outage. The team spent weeks on redundancy, failover and all the operational setup around them. The feature shipped late, the pilot users had moved on, and we eventually shelved it. Those users never needed 99.999%. They needed the feature to exist and to be correct. I had applied the NFR budget of the system we hoped to become to the system we actually were.

Agents make it easier to repeat that mistake, because over-engineered code now costs almost nothing to produce. Ask an agent for "production-grade, highly available" and you'll get retries with jitter, circuit breakers, a multi-region config and a queue you didn't need. Writing the code is no longer the expensive part. Running it is: more infrastructure to pay for, more dashboards, more failure modes, more on-call pages and more to understand at 2 a.m. during an incident. Agents made code cheaper to write, but it costs just as much to operate as it did before.

So keep an explicit "not now" list, and put it in the repo where both agents and humans will read it:

## Deliberately deferred NFRs (docs/architecture/nfr-not-now.md)

| Not now                         | Why                                   | Revisit when                          |
|---------------------------------|---------------------------------------|---------------------------------------|
| Multi-region deployment         | Single-market customers, ops cost     | First contract requiring data residency or >1 region |
| 99.99% availability target      | 99.5% meets current contracts         | Enterprise SLA signed                 |
| Horizontal sharding of orders   | One Postgres instance has headroom    | Sustained DB CPU > 60% at peak        |
| Circuit breakers on internal calls | Few internal hops, simple topology | >3 synchronous internal hops per request |

Agents: do not introduce infrastructure for these items. If a task seems to
require one, stop and flag it in the PR description instead.

Pay attention to the last two lines. They turn "we're not doing that yet" into an instruction an agent can follow and a reviewer can point to. A good revisit trigger is something you can measure: a signed contract, a metric threshold, a user count. "Later" isn't a trigger.

8. Template: the one-page NFR table for your design doc

Put this table in every solution design. If you can't fill a row in completely, that NFR isn't executable yet. Either finish the row or move the NFR to the deferred column on purpose.

| Attribute       | Budget / number                                  | How it's checked                              | Where in the loop      | Owner          | Deliberately deferred until…            |
|-----------------|--------------------------------------------------|-----------------------------------------------|------------------------|----------------|-----------------------------------------|
| Performance     | <= 3 DB queries per request                      | query_budget() in integration tests           | CI (every PR)          | Backend lead   | –                                       |
| Performance     | p95 < 200ms on read endpoints                    | k6 load test vs. reference dataset; SLO alert | Nightly CI + runtime   | Backend lead   | –                                       |
| Reliability     | Mutating endpoints idempotent via Idempotency-Key| Parametrised contract test over all routes    | CI (every PR)          | Payments owner | –                                       |
| Reliability     | 99.5% monthly availability                       | SLO + error-budget alert                      | Runtime                | On-call EM     | 99.99% deferred until enterprise SLA    |
| Security        | Logs contain only allowlisted fields             | AST lint + runtime log filter                 | Pre-commit, CI, runtime| Platform team  | –                                       |
| Security        | No secrets in code                               | gitleaks / secret scanning                    | Pre-commit, CI         | Platform team  | –                                       |
| Observability   | Every request carries request_id end to end      | Contract test on middleware + trace sampling  | CI + runtime           | Platform team  | Full distributed tracing deferred until >3 services |
| Scalability     | –                                                | –                                             | –                      | Architect      | Sharding deferred until DB CPU > 60% at peak |
| Cost (AI/infra) | LLM spend per request < agreed ceiling           | Per-request token metering + budget alert     | Runtime                | EM             | –                                       |

Some tips for using the template well:

• Start with 3-5 rows, not 30. Choose the NFRs whose violations have already hurt you, such as past incidents and review comments that keep coming up. Your postmortems are the richest source.
• Use cheap, deterministic checks early in the loop (lint, pre-commit, CI). Keep the expensive or noisy ones, like load tests and SLOs, for nightly runs and runtime.
• Every failure message should name the NFR, state the budget and suggest the usual fix. Write those messages with an agent as the reader.
• Give every row an owner. A budget nobody owns gets loosened the first time it blocks a release.
• Review the deferred column every quarter. A deferral should be a conscious, dated decision, not somewhere requirements quietly disappear.

9. Common pitfalls

• Budgets without checks. You write "p95 < 200ms" in the doc and stop there. A number that nothing measures is still a vibe, just a more precise one.
• Flaky checks. Put a latency assertion in every PR's CI and it will fail at random. The team, and the agent, will learn to hit retry until it passes. Use deterministic proxies in the fast loop.
• Opt-in coverage. If someone has to remember to write a test for each new endpoint, it will eventually be forgotten. Discover routes, files or services automatically.
• Vague failure messages. "Assertion failed" leaves the agent guessing. Name the NFR, the budget and the fix.
• Agents loosening the constraint. Under pressure, an agent may bump max_queries to 30 or add a field to the allowlist. Protect budget and allowlist files with CODEOWNERS and a branch rule that requires code-owner review, so changing a constraint needs a human decision.
• Enforcing everything. If every best practice becomes a gate, the pipeline gets slow and brittle. Enforce what matters and explicitly defer the rest.

10. Bridge to Day 30

Once your NFRs are executable, the architecture stops being something you check only in design reviews. Every agent and every human checks it on every commit, and nobody has to remember to. That changes what a solution design is: a set of decisions plus the constraints that keep them true. Tomorrow's capstone pulls the whole course into one complete solution design, covering the context, components, trade-offs, the NFR table you built today and the loop that enforces it.

Extend your knowledge

• Read "Building Evolutionary Architectures" by Neal Ford, Rebecca Parsons and Patrick Kua (the 2nd edition adds Pramod Sadalage), especially the chapters on fitness functions. This book popularised architectural fitness functions: architecture checked by automated tests.
• Read the SLO and error-budget chapters of Google's "Site Reliability Engineering" and "The Site Reliability Workbook" to learn how to set runtime budgets you can defend, including when 99.5% is the right answer.
• Study Stripe's public API documentation on idempotent requests. It's a reference design for the Idempotency-Key pattern. Then write the parametrised contract test for your own service.
• Exercise: take your last three incident postmortems and pull one NFR out of each. Fill in a complete row of the NFR table for each one, with a check, a place in the loop and an owner. Then add one deferred row with a measurable revisit trigger.