Your Pipeline Healed Itself Last Night — So Did the Double Charge

Day 11: The Retry Policy Is the Pipeline

Nobody gets paged for a task that succeeds twice. That's exactly the trap. You watch a red box turn green, you close the laptop, you sleep fine — and three weeks later a double-charge ticket lands on your desk with the DAG's name on it. If you've ever flipped on retries and called it 'resilience,' today is the day that word gets cross-examined.

The page that closed itself at 2am

Here's the setup. A payment-write task in the billing DAG times out waiting for an ack from the processor. Airflow marks it failed, the retry policy fires, attempt two runs — and succeeds. The on-call engineer gets the page, watches the task flip from red to green, and goes back to bed. No incident channel. No ticket. This is the pipeline working exactly as designed. Textbook self-healing.

Three weeks later a double-charge ticket lands on the EM's desk. Same customer, same amount, two transactions four seconds apart. Nobody traces it back to that quiet green retry — because nothing about that night looked like a failure. It looked like the system doing its job.

Reconstructing what actually happened

Lay the task open and it's three steps: write the payment row, call the processor, write the ack back. The blip landed between steps two and three. The processor had already charged the card and sent back a success — the response just never made it home before the connection dropped. Airflow saw no ack, a timeout, a failed task, and did precisely what it was told to do: retry with the same input. Attempt two ran all three steps from scratch, including a second call to the processor. Card charged again. This time the ack came back clean.

Notice what didn't go wrong: the orchestrator. It saw a failure signal, ran its configured recovery, exactly as built. The actual bug is a category error — someone treated 'the task failed' and 'nothing happened' as the same fact. They aren't. What really happened was a lost acknowledgment after a side effect that had already, completely, happened.

The actual defect: nobody asked the design question

The defect isn't the retry config, the backoff interval, or the timeout value — you could tune all three for a year and never touch the real problem. The real problem is that when someone flipped retries on for this task, nobody asked the only question that matters: is this task safe to run twice with the same input and the same side effects? Retries went in as an infrastructure setting, a checkbox on the DAG, when they should have been treated as a contract the task itself has to honor. An orchestrator's retry mechanism promises exactly one thing — the task gets attempted again. It promises nothing about what happens when that second attempt collides with the first.

The mental model: three parts, orchestrator gives you one

A task's real reliability spec has three independent parts. Airflow, Dagster, Prefect — pick your orchestrator — hand you exactly one of the three out of the box, and leave you holding the other two, even though 'just enable retries' quietly makes it sound like you're covered on all three.

• ▹Idempotency key — a way for the system to recognize 'this exact unit of work already happened,' so attempt two is a no-op or a safe overwrite instead of a repeat performance.
• ▹Retry/backoff policy — how many times, how far apart, with what jitter. This is the one the config UI hands you for free. It's also, ironically, the least important of the three to get right.
• ▹Failure-detection boundary — the exact line between 'this definitely did not happen' and 'I genuinely don't know.' A timeout on the ack sits on the wrong side of that line more often than people assume — it's proof of nothing, not proof of no-op.

Score the payment task against that list: solid retry/backoff policy, a fuzzy and flat-out wrong failure-detection boundary (timeout read as 'didn't happen'), and zero idempotency key. Two out of three missing is exactly how a green checkmark shipped a bug straight to a customer's statement.

Where this bites hardest today: agents calling tools

Same failure shape, different task name, everywhere in AI systems right now. An LLM agent calling `create_charge` or `send_email` through MCP. An agent framework retrying a tool call because it timed out or the model returned something malformed. A multi-agent pipeline where one agent's output triggers another agent's side-effecting action. Strip the labels off and it's the same DAG-task-with-retries problem — except now the retry decision often gets made implicitly, by the agent runtime or the model itself, with no human ever reading a DAG definition. Skip the idempotency key on your tool schema, let the agent (or its harness) retry on a timeout, and you've built the double-charge incident again, just with an LLM standing in for Airflow's scheduler. The fix doesn't change either: the guarantee has to live in the tool contract, not in the retry wrapper around it.

Fix patterns: safe by construction, not by hope

Tuning the retry count won't touch this. You fix it by changing what the task actually does the second time it runs.

• ▹Idempotency key on every write — generate it upstream (an order ID, or a DAG run ID + task ID + a hash of the input that ignores which attempt this is) and hand it to the downstream system so it can catch a repeat itself.
• ▹Dedup table — before the side effect runs, check a table keyed on that idempotency key. Already there? Skip straight to the ack step, no second charge required.
• ▹Upsert instead of insert — make the write itself idempotent, so running it once or running it ten times lands on the exact same end state.
• ▹Check-then-act, backed by a unique constraint — let the database enforce uniqueness on the idempotency key, so a duplicate write fails fast and loud instead of quietly succeeding twice.
• ▹Separate 'did I send the request' from 'did I get the ack' — persist the fact that the call went out before you sit around waiting on the response, so a retry has something to check before it dials again.

-- before: unsafe by construction
INSERT INTO payments (order_id, amount, status)
VALUES (:order_id, :amount, 'pending');

-- after: safe by construction
INSERT INTO payments (idempotency_key, order_id, amount, status)
VALUES (:idempotency_key, :order_id, :amount, 'pending')
ON CONFLICT (idempotency_key) DO NOTHING;

-- the processor call carries the same key, so a retry
-- either short-circuits on the processor's side or is
-- rejected by the unique constraint before a second
-- charge is ever attempted.

Audit your pipeline today: the 10-minute checklist

This isn't a rewrite project — it's ten minutes. For every task in your DAGs, and every side-effecting tool your agents can call, that has retries enabled, ask one question and actually follow it through:

• ▹List every task or tool with retries > 0 — pull it straight from the DAG configs or the agent tool definitions. Don't trust your memory of 'which ones have retries on.'
• ▹For each one, ask: if this ran twice with the same input, what breaks? Trace the actual side effect — the row written, the email sent, the charge made, the agent action taken — not just what the task returns.
• ▹Check for an idempotency key on the write path. No key means no safety. Flag it and move on.
• ▹Check the failure-detection boundary. Does a timeout get read as 'definitely didn't happen'? If yes, that's a lie the retry policy has been trusting the whole time.
• ▹For agent tool calls specifically: does the schema require or generate an idempotency key, and does the runtime retry on a timeout or a malformed output? Retry with no key — that's the payment bug, just waiting for its turn.

Tomorrow: backfills are retries you asked for

A backfill is a retry wearing a different shirt — you're re-running a task against an input it may have already processed once. The only real difference is who pulled the trigger: you, on purpose, instead of the scheduler, on a timeout. Every idempotency guarantee you build today is the thing standing between tomorrow's backfill and a self-inflicted rerun of this exact incident.

Extend your knowledge

• ▹Stripe's API docs on idempotency keys — the canonical real-world version of this pattern. Worth reading even if you've never touched Stripe.
• ▹Airflow's and Dagster's docs on retry/backoff config — reread them now looking specifically for what they refuse to promise about side-effect safety. It's usually right there, just easy to skim past.
• ▹"Designing Data-Intensive Applications" (Kleppmann) — the chapters on exactly-once semantics and idempotent writes are the theory behind everything in this lesson.
• ▹Pick one production DAG with retries enabled and run today's 10-minute checklist against it before tomorrow's backfill lesson lands.