Physical clock skew

Day 29: Physical clocks lie, so treat 'now' as an interval

Your expiry check is a distributed-systems problem that looks like a one-line if-statement. Expiry, leases, TTLs, rate limits and 'last write wins' all compare a timestamp made on one machine with the clock on another machine. If your code assumes both machines agree on 'now', it works until the day they disagree. Sooner or later they will.

1. The alert

The alert said 'Token used before issued', on about 3% of requests. Every failure came from one node the autoscaler had just added, and only at peak traffic. The rejection log showed the token's iat (issued-at) as 2.4 seconds later than the verifying server's current time. So the server was being handed a token from 2.4 seconds in its own future.

We lost three days. First we blamed the JWT library. Next we blamed the token cache, thinking it was serving stale tokens. After that we blamed the load balancer, in case sticky sessions were sending traffic to the wrong place. Each theory was reasonable, and each one was wrong.

2. The reveal

The new node had booted from an image whose NTP sync hadn't converged yet. Its clock ran a couple of seconds behind the auth service that issued the tokens. The issuer stamped iat with its own 'now'. The verifier compared that iat with its own slower 'now' and concluded the token came from the future.

The code was correct. The bug was an assumption nobody had written down: that both machines agree on 'now'. The autoscaler only added fresh nodes at peak, so the bug only showed up at peak.

3. The lesson in one sentence

On earlier days, Lamport clocks and vector clocks let us order events without trusting any clock. Today is about the clock we still trust: physical, wall-clock time. We rely on it for expiry, leases, TTLs, rate limits and 'last write wins', and it lies to us. Logical clocks tell you which event came first. Physical clocks claim to tell you how much time has passed, but that claim is only roughly true and only within a margin of error.

4. How clocks actually drift

• ▹Quartz drift. A typical server crystal is off by tens of parts per million (ppm), and the error changes with temperature and age. At 20 ppm that's about 1.7 seconds a day. Two machines drifting in opposite directions move apart twice as fast.
• ▹What NTP can promise. In a well-run datacenter or cloud setup, such as chrony pointed at AWS Time Sync or Google's internal NTP, you get offsets in the low milliseconds or better on a good day. Over the public internet you get tens of milliseconds. That number is an estimate, not a guarantee.
• ▹What NTP can't promise. It can't give you a correct clock right after boot, because convergence takes time (that caused our incident). The same goes for after a VM pause or live migration, after a container restores from a snapshot, and when an asymmetric network path skews the offset estimate.
• ▹Leap seconds. Some fleets insert a sudden 61st second. Others smear it across many hours (Google smears over 24). Two machines on different policies can disagree by up to a second during the smear window.
• ▹Steps vs slews. NTP daemons usually slew, meaning they speed the clock up or slow it down gradually. They can also step it, meaning they jump it. A step can move wall-clock time backwards.

Monotonic clocks and wall clocks do different jobs:

• ▹The wall clock (CLOCK_REALTIME, time.time(), Date.now()) answers 'what time is it?'. NTP can step it, including backwards. Use it for timestamps that a person or another machine will read.
• ▹The monotonic clock (CLOCK_MONOTONIC, time.monotonic(), performance.now(), Go's time.Since) answers 'how long has it been?'. It never goes backwards, and its value means nothing on any other machine. Use it for timeouts, latencies, retries and lease durations.
• ▹A real example: during the 2016→2017 leap second, Cloudflare's DNS service hit a negative duration in Go code that assumed time never goes backwards, and the result was an outage. Go later made time.Now() carry a monotonic reading so that durations are safe.

5. Where skew hides in ordinary code

Each of these looks like a local if-statement. Each one actually compares a timestamp from one machine with the clock on another.

• ▹JWT iat/nbf/exp. The issuer stamps the times and the verifier checks them. If the verifier is behind, valid tokens look like they come from the future (our incident). If it's ahead, tokens expire early and users get logged out at random.
• ▹Distributed lock leases. Client A holds a 10-second lease and times it on its own clock. After a GC pause, or with a fast clock, A still believes it holds the lease after the lock server has given it to B. Now two writers are active. This is the core of Martin Kleppmann's critique of Redlock.
• ▹Cache TTLs. The writer sets expires_at = writer_now + 60s, and the reader compares that with reader_now. If the reader's clock is 90 seconds slow, it serves entries that should have expired. If the reader's clock is fast, the cache keeps missing.
• ▹Last-write-wins (LWW) conflict resolution. Replica A writes at 12:00:00.500. Replica B writes later in real time, but B's clock reads 12:00:00.300. B's newer write silently loses, and no error is raised.
• ▹Sorting by created_at across services. An order event from service X sorts before the payment from service Y that caused it, because X's clock runs fast. Audit trails, agent traces and incident timelines all end up in the wrong order.

# LWW done naively: a skewed replica silently loses newer data
def merge(local, remote):
    return remote if remote.ts > local.ts else local   # ts from two different machines

# Lease checked on the client's clock: correct on one machine, unsafe across several
if time.time() < lease.expires_at:   # expires_at was set by the lock server's clock
    write_to_storage(data)           # fencing tokens exist for exactly this case

Agents make this worse. Agent fleets autoscale aggressively, GPU inference nodes get preempted and restored, and sandboxes and microVMs boot from snapshots in milliseconds. Each of those is a new chance to start with an unsynced clock. A multi-agent trace stitched together by timestamp is exactly the 'sort by created_at across services' bug, and it's the trace you'll be reading at 2 a.m.

6. The industry's answers, from cheapest to most rigorous

• ▹1. Leeway margins. Accept a timestamp if it falls within ±N seconds. RFC 7519 explicitly allows 'some small leeway, usually no more than a few minutes' for exp and nbf. It's cheap and solves most of the problem. The cost is that a token stays valid N seconds past its stated lifetime.
• ▹2. Monotonic clocks for durations. Never measure elapsed time by subtracting two wall-clock readings. This rules out negative durations and NTP-step surprises inside a single process. It does nothing for comparisons across machines.
• ▹3. Hybrid Logical Clocks (HLC), from Kulkarni et al., 2014. An HLC timestamp combines physical time with a logical counter (Lamport's idea again). Timestamps stay close to wall-clock time, so people can read them, and they always respect causality: if A happened before B, then HLC(A) < HLC(B), even when clocks are skewed. CockroachDB uses HLCs together with a configured maximum clock offset, and a node shuts itself down if its clock drifts past that offset.
• ▹4. Spanner's TrueTime. TT.now() doesn't return a single point. It returns an interval [earliest, latest] that is guaranteed to contain true time, backed by GPS receivers and atomic clocks. Before acknowledging a commit, Spanner waits until the commit timestamp is definitely in the past on every machine ('commit wait'). That turns uncertainty into a small amount of extra latency, and in return you get external consistency.

# Minimal HLC: physical time + logical counter
class HLC:
    def __init__(self, clock):
        self.clock, self.l, self.c = clock, 0, 0

    def now(self):                      # local event or send
        pt = self.clock()
        if pt > self.l: self.l, self.c = pt, 0
        else:           self.c += 1
        return (self.l, self.c)

    def update(self, ml, mc):           # on receiving a message stamped (ml, mc)
        pt = self.clock()
        new_l = max(self.l, ml, pt)
        if   new_l == self.l == ml: self.c = max(self.c, mc) + 1
        elif new_l == self.l:       self.c += 1
        elif new_l == ml:           self.c = mc + 1
        else:                       self.c = 0
        self.l = new_l
        return (self.l, self.c)

All four answers share one pattern: none of them makes the clock any better. Each accepts that the clock is uncertain and keeps the code correct within a known bound on that uncertainty.

7. The fix we shipped

• ▹A 30-second leeway on the time claims. In golang-jwt v5, WithLeeway applies to exp as well as iat and nbf, so a token can live up to 30 seconds past its stated expiry. Thirty seconds is far more than any skew we expect after sync, and small enough that it doesn't meaningfully extend a token's life.
• ▹A boot gate. The readiness probe fails until chrony reports the node is synced, so the load balancer never sends traffic to a node whose clock hasn't converged.
• ▹A skew metric on every node. node_exporter exposes node_timex_offset_seconds and node_timex_sync_status. We alert when the offset passes a fraction of the leeway or when a node reports unsynced.
• ▹Both clocks in every rejection log line: the issuer's iat/nbf/exp, the verifier's now, and the difference between them. With that line in place, the three-day investigation would have taken minutes.

// golang-jwt/jwt v5
const leeway = 30 * time.Second

tok, err := jwt.Parse(raw, keyFunc,
    jwt.WithIssuedAt(),
    jwt.WithLeeway(leeway),
)
if err != nil {
    claims := unverifiedClaims(raw) // parse without trusting, for logging only
    now := time.Now()
    slog.Warn("jwt rejected",
        "err", err,
        "issuer_iat", claims.IssuedAt,
        "verifier_now", now.UTC(),
        "skew_ms", claims.IssuedAt.Sub(now).Milliseconds(),
        "node", hostname,
    )
}

# readiness gate: don't take traffic until the clock has converged
# waitsync: up to 30 tries, done once the remaining correction is under 100ms
chronyc waitsync 30 0.1 || exit 1

# what to look at when you suspect skew
chronyc tracking          # 'System time' = current offset from NTP time
timedatectl status        # 'System clock synchronized: yes/no'

8. Bridge to agentic engineering: the agent is a node with a frozen clock

A coding agent is like that freshly booted node, except its clock didn't drift by a couple of seconds. It stopped completely at the training cutoff. Its internal 'now' holds whatever library versions, API signatures and best practices were current when the training data was collected. And like our verifier, it acts on that 'now' with full confidence and never questions it.

• ▹The symptom: it calls an API that was deprecated months ago, pins a version with a known CVE, writes config for a flag that has since been renamed, or assumes the current year is the year of its training data. It's 'token used before issued' again: a correct process acting on a wrong 'now'.
• ▹Why it's hard to spot: the output looks right. As with the JWT library, you suspect the code first, when the real cause is the assumption about time.
• ▹The fix is the same one we use for clocks: sync with an authoritative source before acting. Put today's date in the system prompt. Have the agent read the lockfile (package-lock.json, go.sum, uv.lock) before it suggests dependencies. Give it live documentation through a docs-fetching tool or an MCP server instead of relying on what it remembers.
• ▹The boot gate carries over too. In an agent harness, run a 'sync step' at the start of every task (read versions, read the date, check the installed SDK) before any writes are allowed. It's the agent version of readiness waiting on chrony.
• ▹So does the logging rule. When an agent's change fails, log what the agent believed (the version it assumed, the API it called) next to what is actually true (the installed version, the current docs). That gap is the agent's skew. Once you can see it, you can fix it.

On my team, the agent bugs that took longest to find were the ones where the outdated assumption looked like a style choice. The rule is the same as for clocks: don't trust anything's internal 'now', whether it's a node or a model, until it has checked against a current source.

9. Monday checklist

• ▹Grep for timestamp comparisons that cross machines (exp, iat, nbf, expires_at, created_at, updated_at, lease, ttl), anywhere one side came from another service or the database. Every hit needs an explicit skew margin or a written reason why it doesn't need one.
• ▹Add leeway to JWT validation (leeway, clockTolerance or WithLeeway, depending on your library). A few seconds to a few tens of seconds is usually enough. RFC 7519 says 'no more than a few minutes'.
• ▹Use monotonic time for every duration: timeouts, retries, latency metrics, lease countdowns. In code review, reject any code that subtracts two wall-clock values.
• ▹Replace 'last write wins on wall-clock time' with HLCs, version vectors or a single writer. For locks, use fencing tokens instead of trusting lease expiry.
• ▹Gate readiness on NTP sync (chronyc waitsync or systemd-time-wait-sync). Autoscaled, preemptible and snapshot-restored nodes need this most.
• ▹Alert on clock offset per node (node_timex_offset_seconds) at a threshold well below your smallest leeway.
• ▹On every time-based rejection, log both clocks: the remote timestamp, the local now and the difference.
• ▹For agents, put today's date, the lockfile and live docs into context before the agent writes any code.

10. Hand-off to Day 30

For a month we've been learning the same lesson in different forms: every node has its own view of the world. It has its own copy of the data, its own idea of who the leader is, its own picture of membership and, as of today, its own 'now'. Consensus, replication, quorums, logical clocks and physical-clock uncertainty are all tools for building correct behaviour on top of views that disagree. Tomorrow we put the pieces together, for systems made of servers and systems made of agents.

Extend your knowledge

• ▹Martin Kleppmann, 'Designing Data-Intensive Applications' (first edition), Chapter 8 ('The Trouble with Distributed Systems'), the section on unreliable clocks. Read it alongside his blog post 'How to do distributed locking', which covers leases, GC pauses and fencing tokens.
• ▹Corbett et al., 'Spanner: Google's Globally-Distributed Database' (OSDI 2012). Focus on the TrueTime and commit-wait sections.
• ▹Kulkarni et al., 'Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases' (2014), the original HLC paper. Follow it with Cockroach Labs' writing on living without atomic clocks.
• ▹Cloudflare's post-mortem on the leap second at the end of 2016 and the DNS outage it caused. It's a short, concrete case of wall-clock time going backwards in production Go code.