Your "Event-Driven" System Has a Bug Already — It Just Hasn't Fired Yet

Day 10: CDC — The Structural Fix for the Dual-Write Bug You Already Shipped

If your service writes to Postgres and then publishes to Kafka, you already have a bug in production. It just hasn't gone off yet. Change Data Capture (CDC) isn't a data-engineering nicety you bolt on later — it's the structural fix for a category of bug that no amount of code review, testing, or retry logic can eliminate.

Here's the shape of the postmortem that exposes it: an order shows "shipped" in Postgres, but the shipping-notification service never fired, because the Kafka event for that order never arrived. Nobody's job failed. No exception got thrown. The write to the database went through fine. The publish to the queue just... didn't happen. Or happened and got lost. Or happened twice. The line in the postmortem that should worry you is this one: "both writes succeeded, individually, at different times, and nothing in the system ever compared them." That's not an edge case. That's the default failure mode of the architecture.

The pattern hiding under "event-driven"

Almost every team that calls itself event-driven is doing this: write a row to Postgres, then publish a message to Kafka (or SNS, or a webhook, whatever). Two calls. Two networks. Two failure domains. If the process crashes, the pod gets OOM-killed, or the network so much as blips between step one and step two, the database has the truth and the event bus doesn't. Nobody notices until a downstream consumer's numbers stop reconciling — weeks later, when tracing it back is expensive and nobody remembers the deploy that caused it.

async function placeOrder(order) {
  await db.orders.save(order);   // write #1: succeeds
  // <-- crash here, or network partition, or pod eviction -->
  await kafka.publish('orders.created', order); // write #2: never happens
}

Why this isn't a bug in your code — it's a category error

You cannot make two independent writes to two independent systems atomic without a distributed transaction spanning both — and nobody runs XA/2PC across Postgres and Kafka in production, for good reason: it couples their availability, tanks throughput, and turns a queue outage into a database outage. So teams either eat the drift silently, or bolt on an outbox table plus a relay poller to paper over it. That's an improvement, sure, but it's still two logical writes dressed up as one. The fix isn't a smarter retry policy. It's removing the second write, full stop.

What CDC changes structurally

CDC makes the database's write-ahead log (WAL) the only write that happens. The event stream gets derived by reading that WAL after the fact — it's not a second write your application code performs. There's no "crash between step one and step two," because there is no step two. If the transaction commits, it's in the WAL, and the WAL is what the event stream reads from. Drift isn't rare with CDC. It's structurally impossible, because there's exactly one source of truth and exactly one write path into it.

The mechanics, briefly

Postgres already writes every committed change to its WAL before that change is visible anywhere else — that's how it survives crashes and does replication. A logical decoding plugin turns those WAL entries into a structured change stream: pgoutput, built into Postgres core and Debezium's default since v1.6, or the older wal2json extension, which emits JSON directly. Debezium is the connector — usually run on Kafka Connect — that consumes that decoded stream and turns it into per-table topic messages, one topic per table, one message per row change, in commit order. If you've followed the earlier lessons on logs-as-source-of-truth and log-structured storage, you already know this shape: the WAL is the log, Debezium is the tailer, and Kafka is just another consumer reading a log that already existed.

What CDC does NOT fix

CDC kills dual-write drift. It does nothing about the distributed-systems reality that lives downstream of the WAL:

• ▹Consumer lag — a consumer that falls behind means downstream state is stale even though the source of truth is correct; CDC buys you eventual correctness, not latency.
• ▹Schema evolution — rename a column or change its type in Postgres and you've shipped a breaking change to every downstream parser reading that topic. You still need a schema registry and a compatibility policy; CDC won't invent one for you.
• ▹Ordering across partitions — Debezium holds per-table (usually per-key) ordering, but the moment you fan out to multiple partitions or topics, cross-entity ordering stops being free.
• ▹Exactly-once delivery to consumers — CDC gives you exactly-once capture off the WAL, but your consumers still need idempotent handling for their own retries and rebalances.

Why this matters more, not less, in the AI era

The dual-write bug is quietly multiplying right now, because AI-era systems added a new favorite second write: syncing the vector index. "Save the document, then upsert the embedding into Pinecone/pgvector/Weaviate" is the exact same anti-pattern as "save the order, then publish to Kafka" — except now the drift shows up as an agent confidently answering from a stale or missing embedding, which is a far harder thing to notice than a missing shipping notification. CDC into the embedding pipeline (WAL → Debezium → re-embed job) closes that gap the same way it closes the Kafka gap. And watch what your code-generation agents actually produce here: LLMs were trained on codebases full of save()-then-publish() code, so they'll reproduce the anti-pattern by default unless your review process or lint rules specifically catch it. A category error doesn't stop being one just because an agent typed it instead of a human.

Audit prompt

Grep your own services for the pattern: save( followed shortly by publish( or emit( or produce( in the same function, with no outbox table and no CDC connector in between. That line is exactly what CDC replaces. Tomorrow picks up where the WAL's guarantees stop — schema evolution, and the registry you need once every consumer depends on a format that WAL-derived events can silently change.

Extend your knowledge

• ▹Read the Debezium docs' "How the PostgreSQL connector works" page to see exactly what wal2json/pgoutput output looks like before it hits Kafka.
• ▹Look up the transactional outbox pattern (as documented by Chris Richardson's microservices.io) — it's the manual, pre-CDC version of the same fix, and worth understanding to see what CDC is automating away.
• ▹If you're running a Postgres-to-vector-store sync today, check whether that sync is a second application write or already CDC-driven. It's the same audit as the Kafka case.
• ▹Preview for tomorrow: search for how your team handles schema compatibility on Kafka topics (Confluent Schema Registry, or the lack of one) — that's exactly where the WAL's guarantees stop and your discipline has to start.