Pub/sub hidden contracts

Day 25: Pub/Sub and the Contract Nobody Wrote Down

Why this matters: an agent renames one field. Every test passes, a reviewer approves it in four minutes, and three downstream services stay quietly wrong for two days. Pub/sub is built to hide who depends on your events. Coding agents only reason about what they can see. Together, those two facts give you changes that pass every check and still break production. Today you'll learn why that happens and how to make the hidden dependencies visible to agents and reviewers.

1. The four-minute approval

This scenario is a composite, but I've seen versions of it play out more than once. An engineer asks a coding agent to tidy up the order service. As part of the cleanup, the agent renames a field on the OrderPlaced event from `total` to `totalAmount` to match the new domain model. It updates the publisher, the TypeScript type, the serializer and the unit tests. The tests pass and CI goes green. The diff is clean and well commented. A reviewer approves it in four minutes.

Two days later, three things are broken. The loyalty service has been awarding 0 points, because `event.total` is undefined and its code falls back to 0. The analytics pipeline has been writing nulls into the revenue column. The fraud scorer has been skipping its high-value check, because `undefined > 50000` is false. Nothing threw an error and nobody got paged.

So who was supposed to know? The agent couldn't see those consumers. The reviewer didn't remember them, and nothing in the repo listed them. Nobody did anything wrong. The architecture did exactly what it was designed to do, and that's the problem.

2. Today's piece in one picture

Pub/sub (publish/subscribe) is a messaging pattern. A publisher sends a message to a topic, and every subscriber on that topic gets its own copy. The publisher doesn't know how many subscribers there are or who they are. There could be zero or forty.

• ▹Point-to-point queue (Day 24): each message goes to one logical consumer. You're spreading work across workers, and the producer and consumer usually know about each other.
• ▹Pub/sub: each message fans out to N subscribers. The message is a broadcast fact ("an order was placed"), not a command ("process this order").
• ▹The property that matters here is that the publisher is deliberately blind to its consumers. That's a design goal, not a gap. Kafka topics with multiple consumer groups, SNS → SQS fan-out, Google Cloud Pub/Sub, NATS subjects and EventBridge rules all work this way.

3. The trade you signed up for

Decoupling buys you something real. A new team can subscribe to orders.placed without filing a ticket with the order team, without a deploy, and without the publisher changing a line of code. That's why event-driven architectures scale with the organisation as well as with traffic.

The cost is that the call graph is no longer in the code. In a monolith or a direct RPC call, `loyalty.awardPoints(order.total)` is a reference your IDE can follow. Put a broker in the middle, and the only link between `publish('orders.placed', event)` and `event.total` in another repo is a topic name in a config file. You can't run "Find usages" across a broker. The dependency still exists, but your tools can't see it.

So the real contract in pub/sub isn't the message schema. It's the set of unwritten assumptions about who's listening and what each listener relies on.

4. Why humans mostly got away with it

Before agents, teams kept the dependency graph in people's heads. "Ask Minh before you touch that event." A pinned Slack thread from 2022. The senior engineer who reads the diff and says, "Wait, analytics reads that field." Tribal knowledge is fragile, but it worked as a slow, informal check. A change to a widely used event usually went past someone who remembered the consumers.

Two things are eroding that check. Code changes faster now, so more payload changes reach review. And the change is more and more often proposed by an agent, which has never seen the Slack thread and has never met Minh.

5. Why agents can't get away with it

A coding agent reasons from its context window: the files it opened, the search results it got, the test output it saw. When it renames a field, its evidence that the change is safe is that the type checker and the tests pass. As far as the type checker knows, the publisher is a closed world. The event type is defined in this repo and only used in this repo, so the change looks perfectly consistent.

I call this "locally correct, globally wrong." Within what the agent can observe, the change is flawless. The coupling that matters is outside its view, in other repos, in other teams' consumer configs, in some dbt model.

Multi-agent systems research describes this precisely. Agents act on the environment they can observe, and how well they coordinate is limited by what that environment exposes. This is the partial-observability problem behind much of the decentralised coordination literature. Pub/sub is an environment designed to hide the coupling, so an agent coordinating with downstream consumers it can't perceive will break them. That's an information failure, not a reasoning failure. You fix it by changing the environment. A cleverer prompt won't help.

6. The three kinds of hidden contracts that break

• ▹Field shape: renaming, removing or retyping a field, such as `total` → `totalAmount`, or `customerId` changing from string to number. This is the easiest kind to detect.
• ▹Semantics: the name and type stay the same, but the meaning changes. `total` switches from cents to dollars. `status: "placed"` now also covers orders waiting on fraud review. `timestamp` moves from created-at to persisted-at. Every schema check passes, and the numbers are quietly off by a factor of 100.
• ▹Timing and ordering: a subscriber assumed the event fires after payment is captured, and the agent moved the publish earlier to cut latency. Or a consumer assumed OrderPlaced always arrives before OrderShipped for the same order, and a partition-key change broke that. The payload never changed.

Schema registries (Confluent Schema Registry, AWS Glue Schema Registry, Apicurio) with compatibility modes catch only the first kind, and only when the compatibility mode is strict enough. You need one, but it won't catch everything. Agents are especially likely to make the second and third kinds of change, because those look like refactors or performance improvements.

7. Fix 1: the subscriber registry

Make subscriptions easy to find. Keep one small YAML file per topic in the publisher's repo, next to the event definition. For each consumer, it records the owner, the fields it reads and what it assumes. It doesn't need to be clever. It just needs to be where the agent will look.

# events/orders.placed/subscribers.yaml
topic: orders.placed
publisher: order-service
owner: team-checkout
subscribers:
  - name: loyalty-service
    owner: team-growth
    endpoint: arn:aws:sqs:ap-southeast-2:123456789012:loyalty-orders
    reads: [orderId, customerId, total]
    assumptions:
      - "total is integer cents, tax-inclusive"
      - "event fires only after payment is captured"
    on_missing_field: "defaults to 0 (silent!)"
    contract_test: pacts/loyalty-service-order-service.json

  - name: analytics-ingest
    owner: team-data
    endpoint: arn:aws:sqs:ap-southeast-2:123456789012:analytics-orders
    reads: [orderId, total, currency, placedAt]
    assumptions:
      - "placedAt is UTC ISO-8601, set at order creation"
    on_missing_field: "writes null"

  - name: fraud-scorer
    owner: team-risk
    endpoint: arn:aws:sqs:ap-southeast-2:123456789012:fraud-orders
    reads: [orderId, customerId, total, shippingAddress.country]
    assumptions:
      - "at most one OrderPlaced per orderId"
      - "total > 50000 cents triggers manual review"
    on_missing_field: "skips check (silent!)"

Three fields matter most. `reads` answers "will my rename hit you?" `assumptions` records the semantic and timing contracts a schema can't express. `on_missing_field` tells the reviewer whether a break will be loud or silent. If it's silent, the bar for approval should be higher.

Then make this the first file the agent is told to read when it touches a publisher. Fix 3 covers how.

8. Fix 2: consumer-driven contract tests

The registry tells the agent what might break. Contract tests make the build fail when it does. With consumer-driven contracts, each subscriber writes down the message shape it depends on. That file, called a pact, is published to a broker. The publisher's CI then checks that the messages it actually produces still satisfy every pact. Pact supports asynchronous messages as well as HTTP.

Consumer side (in loyalty-service):

// loyalty-service/test/orderPlaced.pact.test.js
const { MessageConsumerPact, Matchers, asynchronousBodyHandler } = require("@pact-foundation/pact");
const { string, integer } = Matchers;
const { handleOrderPlaced } = require("../src/handlers");

const pact = new MessageConsumerPact({
  consumer: "loyalty-service",
  provider: "order-service",
  dir: "./pacts",
});

describe("orders.placed as consumed by loyalty-service", () => {
  it("awards points from an OrderPlaced event", () =>
    pact
      .expectsToReceive("an OrderPlaced event")
      .withContent({
        orderId: string("ord_123"),
        customerId: string("cus_42"),
        total: integer(4999), // cents
      })
      .withMetadata({ topic: "orders.placed" })
      .verify(asynchronousBodyHandler(handleOrderPlaced)));
});

Provider side (in order-service CI). This test builds the real event through the real code path and checks it against every consumer pact:

// order-service/test/orderPlaced.provider.pact.test.js
const { MessageProviderPact } = require("@pact-foundation/pact");
const { buildOrderPlacedEvent } = require("../src/events");
const { sampleOrder } = require("./fixtures");

describe("order-service honours its subscribers", () => {
  it("satisfies all consumer pacts for orders.placed", () =>
    new MessageProviderPact({
      provider: "order-service",
      pactBrokerUrl: process.env.PACT_BROKER_URL,
      providerVersion: process.env.GIT_SHA,
      publishVerificationResult: process.env.CI === "true",
      messageProviders: {
        "an OrderPlaced event": () => buildOrderPlacedEvent(sampleOrder),
      },
    }).verify());
});

With this in place, the agent's rename fails provider verification because loyalty-service expected `total` and didn't get it. The agent sees a red test that names the consumer, which is exactly the signal missing from its context. Add Pact's `can-i-deploy` check to your pipeline so a broken contract also blocks the deploy.

Contract tests have limits. They catch shape changes reliably. They catch semantic changes only if the consumer encodes the meaning, for example by asserting cents-scale values in the handler test or using a stricter matcher. They don't catch timing or ordering changes at all. For those, you still rely on the `assumptions` in the registry and on human review.

9. Fix 3: SUBSCRIBERS.md plus one agent rule

The YAML is for machines and CI. Humans, and the agent on its first read, need something easier to scan. Put a short SUBSCRIBERS.md next to each topic's event definition and generate it from subscribers.yaml so the two never disagree. Then add one line to your agent instructions file (CLAUDE.md, AGENTS.md, .cursor/rules, or Copilot's instructions file):

## Events
- Before changing any event payload, publish timing, or partition key, read SUBSCRIBERS.md for the topic and list affected consumers in the PR.

The rule covers publish timing and partition keys as well as payloads, because that's where the third kind of hidden contract lives. Pair it with a PR template section so the agent's output has a fixed format that a reviewer can check in seconds:

## Event contract impact
Topic(s) touched: `orders.placed`
Change type: [x] field shape  [ ] semantics  [ ] timing/ordering  [ ] none

| Subscriber | Owner | Reads changed field? | Failure mode | Action |
|---|---|---|---|---|
| loyalty-service | team-growth | yes (`total`) | silent: defaults to 0 | dual-publish `total` + `totalAmount` for 2 releases |
| analytics-ingest | team-data | yes (`total`) | silent: writes null | ticket DATA-412 to migrate dbt model |
| fraud-scorer | team-risk | yes (`total`) | silent: skips check | blocked until team-risk acks |

Contract tests: provider verification [x] passing  
Owners notified: @team-growth @team-data @team-risk

Now replay the OrderPlaced scenario with all three fixes in place. The agent reads SUBSCRIBERS.md first, sees three consumers reading `total`, and proposes an expand-and-contract change: add `totalAmount`, keep `total`, and remove `total` later. If it renames the field anyway, provider verification fails. If the change somehow gets past both, the reviewer sees three "silent" failure modes in the PR table. A four-minute approval is no longer possible, and that's the goal.

10. What this costs and when it's worth it

The main cost is registry drift. Someone adds a subscriber in Terraform and never updates the registry, so the agent ends up trusting an out-of-date map. A stale registry can be worse than having none, because it creates false confidence. Have CI catch drift by comparing the broker's live subscriptions against the registry and failing when they differ.

#!/usr/bin/env bash
# ci/check-subscribers.sh <topic-arn> <subscribers.yaml>
set -euo pipefail
TOPIC_ARN="$1"; REGISTRY="$2"

live=$(aws sns list-subscriptions-by-topic --topic-arn "$TOPIC_ARN" \
  --query 'Subscriptions[].Endpoint' --output text | tr '\t' '\n' | sort)
registered=$(yq '.subscribers[].endpoint' "$REGISTRY" | sort)

unregistered=$(comm -23 <(echo "$live") <(echo "$registered"))
if [ -n "$unregistered" ]; then
  echo "Subscribed but not registered in $REGISTRY:"
  echo "$unregistered"
  exit 1
fi

The same approach works on other brokers. Use `gcloud pubsub topics list-subscriptions` on Google Cloud, consumer-group listings on Kafka, and rule targets on AWS EventBridge. Run the check on a schedule. Also run it in the infra repo whenever a subscription changes, so whoever adds a consumer has to register it. That puts ownership with the subscriber, which is the only party that really knows its own assumptions.

• ▹My rough threshold: with fewer than about 3 consumers per topic, all owned by teams that talk every day, a comment above the event type listing the consumers may be enough. Agents read comments in the files they open.
• ▹With 3 or more consumers, or any consumer owned by another team, add the registry and SUBSCRIBERS.md.
• ▹Add contract tests when a consumer fails silently or touches money, risk or compliance. Loud failures (crashes, dead-letter queues) are cheaper to find after the fact.
• ▹Skip the registry for a topic whose only consumer is in the same repo. Find usages still works there.

11. Monday checklist

• ▹Pick the topic with the most subscribers. If you don't know which one that is, you've found your first gap.
• ▹Write its subscribers.yaml and SUBSCRIBERS.md. Ask each consumer team three questions: which fields do you read, what do you assume about meaning and timing, and what happens when a field is missing?
• ▹Add one consumer-driven contract test and verify it in the publisher's CI. Start with the consumer whose failure is the most silent.
• ▹Add the one-line agent rule to CLAUDE.md or AGENTS.md, plus the PR template section.
• ▹Replay the OrderPlaced scenario by asking your agent to rename a field on that topic. Check that it reads SUBSCRIBERS.md and lists the consumers. If it goes ahead with the rename anyway, check that a red contract test stops it.
• ▹Set up the drift check so the registry can't go out of date without CI flagging it.

12. Bridge to Day 26

Once you know who's listening, the next question is what each listener does when the same message arrives twice. Pub/sub brokers redeliver on timeouts, rebalances and retries. A fraud scorer that assumes "at most one OrderPlaced per orderId" depends on a guarantee most brokers don't make. Tomorrow covers exactly-once myths and how to build idempotent consumers. It also covers why agents tend to write handlers that work the first time a message arrives and break the second time.

Extend your knowledge

• ▹Pact's documentation on message pacts (contract testing for asynchronous, event-driven systems) and the Pact Broker's `can-i-deploy` tool. Start at docs.pact.io.
• ▹Martin Fowler's article "What do you mean by 'Event-Driven'?" It separates event notification, event-carried state transfer, event sourcing and CQRS, and each of these creates different hidden contracts.
• ▹The AsyncAPI specification, for describing your topics and their messages in a machine-readable format you can generate SUBSCRIBERS.md from.
• ▹Confluent's documentation on Schema Registry compatibility modes (BACKWARD, FORWARD, FULL and their TRANSITIVE variants), so you know exactly which field-shape changes your registry already blocks.