Posts tagged Data Engineering — AI × software engineering, made practical.
A scheduling agent swore up and down it was March in September — and the root cause wasn't a bad model, it was an idempotency key that never found out the function behind it had changed.
The real reason Parquet screams isn't zstd or snappy — it's a footer trick that lets the engine refuse to read most of your data at all, and if you don't respect its shape, you can make Parquet slower than the CSV you replaced.
The tenant_id shard that held up for a year under real human traffic didn't survive three weeks of an unattended AI agent — here's the layer that would have turned the outage into a config change.
Why a full Iceberg or Delta migration can leave the exact same team gatekeeping every schema change eighteen months later — and the one blunt question that exposes it in ten seconds.
An agent-assisted dbt refactor passed every test, shipped clean, and quietly inflated regional revenue 2.1x — here's the one-line change that caused it and the test nobody had written to catch it.
An agent pulled a customer's lifetime value from the one table every analyst on the team had learned, by word of mouth, never to touch — this is the lesson on why schema-on-read was always a deal with humans, and what it takes to make that deal legible to a machine.
A gut-check for engineers who reach for Kafka by default: if the consumer can't perceive the latency win, streaming isn't buying speed, it's buying you an incident channel.
The LLM call you're treating as "just reasoning" is doing the exact job a dbt model does, except nobody's diffing it, versioning it, or noticing when ten agents quietly disagree on what "churn" means.