Write-ahead logging for coding agents

Run an agent on a long enough task and sooner or later it dies partway through. The process gets killed, you hit a rate limit, the laptop goes to sleep. Afterwards, what's on disk is either a clean point to resume from or a branch you end up deleting. One design choice decides which, and it has nothing to do with the model or the size of its context window.

The hook: dead at file 23 of 40

You ask an agent to rename `getUserById` to `fetchUser` across 40 files. It works through them one at a time. At file 23 the process dies. You restart and type "continue". The model looks at the repo, sees some files with the old name and some with the new one, and guesses. It redoes files 19 to 22, which were already fine. It skips 23 because the file looks half-renamed and it concludes that one is done. Then it picks up at 24. Now the diff compiles in some places and fails in others. Nowhere is there a record of what the agent meant to do. You only have what it happened to do.

I've cleaned up branches like this more times than I want to count. Every time, someone suggested a bigger context window or a better summary. Neither fixes it.

The real problem: no durable record of intent

A running agent holds two kinds of state:

• ▹**What happened** lives in the repo: files on disk, git objects, the database. It survives a crash.
• ▹**What the agent meant to do** lives in the context window. That means the plan, the step it was on, and the edit it was about to make. All of that dies with the process.

So after a crash you're left with the effects and none of the intent. If you ask the model to rebuild intent from a half-changed repo, you're asking it to do forensics, which is not the same as recovery. A larger context window can't help, because the context window is exactly what got lost. You need a record of intent written somewhere durable *before* the side effect happens.

The WAL rule in one sentence

**Log first, apply second, never the other way round.** That's all the write-ahead log (WAL) rule says. Databases have relied on it for decades. PostgreSQL uses it, SQLite does in WAL mode, and so does almost every storage engine you've worked with.

Take a bank transfer of $100 from A to B. Debit A, crash before crediting B, and $100 has disappeared. Nothing tells you whether the transfer was supposed to finish. With a WAL, you first write `T42: debit A 100, credit B 100` to the log and force it to disk. Only after that do you touch the balances. When you're done, you write `T42: commit`. If you crash in the middle, the log still says what T42 was meant to do, so you can finish it. The balances on their own could never tell you that.

You've met this log earlier in the series. Raft's replicated log and Kafka-style append-only logs are the same structure: an ordered, durable list of intended changes. Replication uses that list to get many nodes to agree. A WAL uses it to let one node recover after a crash.

WAL anatomy, kept minimal

• ▹**LSN (log sequence number):** an ever-increasing number that identifies each intent. It puts everything in a total order and lets you say "everything up to N is done". In this post, a commit marker carries the LSN of the intent it closes.
• ▹**Intent record:** what you're about to do, written in enough detail that you could do it again without the model. "Fix the file" isn't enough. You want "in file Y, replace symbol X with Z; the file currently hashes to H1 and should hash to H2 afterwards".
• ▹**Apply:** the side effect itself, like writing the file or running the command.
• ▹**Commit marker:** a second record saying the intent at LSN N has landed and been verified.
• ▹**fsync:** the moment a record becomes real. Until the bytes reach stable storage, a crash can make the record disappear. A `write()` that has only reached the OS page cache is a promise, not a fact.

import json, os

def log_append(path, record):
    # One JSON object per line. Not durable until fsync returns.
    with open(path, "a", encoding="utf-8") as f:
        f.write(json.dumps(record) + "\n")
        f.flush()
        os.fsync(f.fileno())

Here's one step laid out on a timeline, with what recovery does for a crash at each point:

time ------------------------------------------------------------------------->

 [1] append intent 24 -> [2] fsync -> [3] edit file -> [4] verify -> [5] append commit 24 -> [6] fsync

crash before [2]      intent may or may not have reached disk (a torn
                      last line is discarded). File untouched either way.
                      Recovery: nothing to undo. Step 24 is simply next.

crash in [2]..[3)     intent durable, file hash == pre.
                      Recovery: REDO intent 24.

crash during [3]      file may be torn (half-written).
                      hash != pre and hash != post.
                      Recovery: restore file from checkpoint, then REDO.

crash in (3]..[6)     file hash == post, but no durable commit.
                      Recovery: re-verify, write commit 24, move on.

crash after [6]       committed.
                      Recovery: skip. Resume at 25.

There's a clear answer for every gap, and that's what writing intent before the effect buys you. If you have effects and no intent, every gap looks identical: "something might have happened".

Recovery is replay, and replay needs idempotency

In this model, recovery means **redo**: find each intent without a commit marker and apply it again. That's only safe when applying an intent twice gives the same result as applying it once, which is what idempotent means. You often can't tell whether the first attempt landed, so the action has to be safe either way.

# SAFE to replay: describes an end state
{"tool":"replace_symbol","file":"src/api/users.ts","old":"getUserById","new":"fetchUser"}
  run once  -> renamed
  run twice -> no matches left, no-op

{"tool":"replace_function","file":"src/billing/tax.ts","name":"calcTax","body":"<full new body>"}
  run twice -> body replaced with itself, no-op

# UNSAFE to replay: describes a delta
{"tool":"append_line","file":".env.example","line":"FEATURE_FETCH_USER=1"}
  run twice -> the line appears twice

{"tool":"shell","cmd":"sed -i 's/User/UserV2/g' src/models.ts"}
  run twice -> UserV2V2

# FIX: turn the delta into an end state
{"tool":"ensure_line","file":".env.example","line":"FEATURE_FETCH_USER=1"}
{"tool":"write_file","file":"src/models.ts","post":"sha256:7c2e..."}

Most agent setups break right here, because agents reach for deltas: `sed`, `>>`, "insert after line 40", `git commit -am`, `npm version patch`. Each one makes sense on its own, and each one does something different the second time it runs. If your agent's tools are mostly deltas, a WAL will replay them exactly as logged and break the repo exactly as logged. Give the agent tools that describe **end states**: replace this symbol, set this file to this content, make sure this line exists. Log the expected post-image hash too, so the harness can check whether the edit landed.

Torn writes are a related trap. If the agent writes a file in place and dies halfway through, you're left with a file that is neither the old version nor the new one. Instead, write to a temp file, fsync it, and `rename()` it over the original. On POSIX filesystems a rename within the same filesystem is atomic, so the file is always either the pre-image or the post-image. Skip the fsync, though, and some filesystems can leave you an empty or partial file after a power loss, even though the rename was atomic.

Mapping it to the agent

Any tool call that changes the world, whether it's a file edit, a shell command, a migration, or an API call, gets a JSON intent line **before** it runs and a commit line **after** it's verified. The harness writes these lines. The model doesn't. The model proposes an action, and the harness logs it, applies it, checks it, and commits it. Here's the tail of the log for the 40-file rename, where LSN 1 is the plan and LSN N+1 is file N:

{"lsn":1,"type":"plan","task":"rename getUserById -> fetchUser","files":40,"base":"git:4e1a2c7"}
{"lsn":22,"type":"intent","tool":"replace_symbol","file":"src/api/orders.ts","old":"getUserById","new":"fetchUser","pre":"sha256:a41f...","post":"sha256:0be9..."}
{"lsn":22,"type":"commit","verified":"hash==post && tsc --noEmit ok"}
{"lsn":23,"type":"intent","tool":"replace_symbol","file":"src/api/sessions.ts","old":"getUserById","new":"fetchUser","pre":"sha256:5d10...","post":"sha256:c3a7..."}
{"lsn":23,"type":"commit","verified":"hash==post && tsc --noEmit ok"}
{"lsn":24,"type":"intent","tool":"replace_symbol","file":"src/api/users.ts","old":"getUserById","new":"fetchUser","pre":"sha256:91ee...","post":"sha256:7c2e..."}

The last line shows where it died. Intent 24 (file 23) is durable and has no commit. `replace_symbol` is deterministic, so the harness worked out the `post` hash before it touched the file. Because of that, recovery is a hash comparison and nobody has to guess.

The replay loop: the harness recovers, not the model

On restart, the model shouldn't run first. The harness reads the log and works out the exact state. Only then does it give the model a precise point to resume from.

def recover(wal_path, repo, checkpoint_ref):
    recs = [json.loads(l) for l in open(wal_path)]
    committed = {r["lsn"] for r in recs if r["type"] == "commit"}
    pending = [r for r in recs if r["type"] == "intent" and r["lsn"] not in committed]
    last_ok = max(committed, default=0)

    for it in pending:  # normally exactly one
        h = sha256_file(repo / it["file"])
        if h == it["post"]:
            verify(it); log_append(wal_path, {"lsn": it["lsn"], "type": "commit"})
            status = "applied, commit was missing; now committed"
        elif h == it["pre"]:
            status = "not applied; redo"
        else:
            git_restore(repo, checkpoint_ref, it["file"])
            status = "partially applied; restored from checkpoint; redo"
        yield it, status

    # The harness builds the resume prompt from facts, not from a summary.

Put the two resume prompts side by side. Here's the usual one:

You were interrupted. Summarise what you were doing and continue.

And the one built from the WAL:

Task: rename getUserById -> fetchUser (plan LSN 1, 40 files, base 4e1a2c7).
Committed through LSN 23 (files 1-22). Do not touch them.
LSN 24 (src/api/users.ts) was intended, not committed. File hash matched
neither pre nor post: partially applied. Harness restored it from 4e1a2c7.
Action: redo LSN 24, then continue at LSN 25 (file 24).

The first prompt hands recovery to the model. The second treats recovery as finished and asks the model for the next piece of work. Models are good at the work and unreliable at forensics, so give them the second prompt.

Checkpoints and truncation

A WAL only ever grows, and so does an agent transcript. Leave it alone and it gets slow to replay and painful to read. Databases deal with this using **checkpoints**. At a known-good point, you make sure every effect up to LSN N is durable on its own, record that fact, and then throw away (or archive) the log entries before N.

• ▹For a coding agent, the obvious checkpoint is a **git commit** on the agent's branch at a safe point, say after a batch of files is done and the typecheck passes.
• ▹Write a checkpoint record such as `{"lsn":41,"type":"checkpoint","git":"b81d0fa"}`. From then on, recovery only reads records after LSN 41, and torn files get restored from `b81d0fa`.
• ▹Archive everything earlier. Keep it for auditing if you want, but recovery won't read it again.

This is also why **compacting context without a checkpoint is dangerous**. Auto-compaction in tools like Claude Code summarises older turns to free up tokens. That works when everything before the summary already exists in durable state, meaning a git commit plus a truncated WAL. When it doesn't, compaction deletes the only copy of intent and keeps a lossy paraphrase in its place. So checkpoint first and compact second. Truncating the log and compacting the transcript are the same kind of operation, and neither is safe without a checkpoint behind it.

Where it breaks

Some tool calls have external side effects that no amount of tool reshaping will make idempotent: sending an email, charging a card, posting to Slack, running a migration that drops a column. Replaying one of those doesn't recover anything. It causes a second incident. The standard ways to handle them:

• ▹**Idempotency key.** Derive a stable key from the log, such as `run_id:lsn`, and send it with the request. Services built for this return the original result instead of acting twice. Stripe's `Idempotency-Key` header is the best-known example. If the downstream API supports keys, this is the cleanest fix.
• ▹**Confirm-before-commit (in-doubt state).** With no keys available, recovery can't redo blindly. Mark the intent *in doubt*, then check the outside world: is the email in the sent folder, did the migration version table move? Commit or redo only after that. For anything irreversible, put a human approval step between intent and apply.
• ▹**Split reversible from irreversible.** Batch the irreversible actions at the end, behind a checkpoint. A crash anywhere in the reversible part then can't reach them.

Be clear about what a WAL actually buys you. It makes recovery *possible*: after a crash you can get back to exactly the state the agent intended. It does nothing to make that intent *correct*. If the plan at LSN 1 was wrong, the WAL will help you finish the wrong plan with great reliability. Correctness still comes from review, tests, and verification steps. The WAL's job is to stop a crash from scrambling those signals.

Practical checklist

• ▹**Every intent record needs five fields:** `lsn` (ordering), `tool` + args (what to do, replayable without the model), `target` (file, table, or endpoint), `pre` (state hash or precondition before), `post` (expected state hash or check after). For external calls, add an idempotency key derived from `run_id:lsn`.
• ▹**Keep the log out of the context window and out of the working tree**, e.g. `~/.agent-runs/<run_id>/wal.jsonl`. Inside the repo, it can be wiped by `git clean -fdx`, `git checkout .`, or the agent's own `rm -rf`, and it can get committed by accident.
• ▹**fsync after every append.** Until it's fsynced, the log is only a draft.
• ▹**"Committed" means verified, not "the tool returned".** The post hash matches and a check passed: typecheck, test, or a read-back query. An exit code of 0 proves nothing.
• ▹**Use end-state tools instead of delta tools.** `replace_symbol`, `write_file` with a post hash, and `ensure_line`, not `sed`, `>>`, or "insert after".
• ▹**Checkpoint to git at safe points, then truncate.** Only compact the transcript once a checkpoint is in place.
• ▹**The one-line test:** `kill -9` your agent at a random step, restart it, and diff the result against a clean run. Any difference means you don't have recovery yet.

Bridge to Day 29

One agent with one log can recover, because there's exactly one order of events: the order of its LSNs. Now put two agents on the same repo, one renaming symbols and one upgrading a dependency. Each has its own log and its own view of what happened first. Their commits interleave, and replaying either log by itself produces a state the other agent never saw. So how do separate writers agree on a single order for each other's commits? That's Day 29.

Extend your knowledge

• ▹Read 'Reliability and the Write-Ahead Log' in the PostgreSQL documentation. It's a short, concrete look at how WAL, fsync, and checkpoints work in a production system.
• ▹In Martin Kleppmann's 'Designing Data-Intensive Applications', read the storage engines chapter ('Storage and Retrieval') and the transactions chapter. Together they cover WAL, crash recovery, and atomicity without heavy database internals.
• ▹Skim the ARIES paper (C. Mohan et al., 1992), the classic redo/undo recovery design that most database WALs come from. Focus on what LSNs and checkpoints do.
• ▹Look at durable execution engines such as Temporal, where replaying the event history is the same pattern applied to workflows. Then wrap one of your agent's file-edit tools in intent and commit logging and run the kill -9 test.