Your Parquet Query Is 44x Faster Than CSV — and Compression Has Almost Nothing to Do With It

Day 9: The Two-Level Skip Trick That Makes Parquet Fast

You've queried Parquet a thousand times through Spark or DuckDB and never once opened one. Fair — most people haven't. But here's the thing that changes how you think about it: Parquet isn't fast because it's compressed. It's fast because of what it refuses to read. Those are two very different engineering claims, and only one of them explains why your pipeline flies instead of just shrinking.

90ms vs 4s, same data, same filter

Take a table of agent run logs — 50M rows. One column is trace_id, one is latency_ms, one is a fat raw_response text blob. Run `WHERE latency_ms > 5000` against the CSV: about 4 seconds, because every engine has to walk every row and decode every field just to check a single number. Run the same filter against the Parquet version: about 90ms. No compression codec on earth explains a 44x gap like that. The engine didn't decompress faster — it skipped almost everything and never touched it.

This is not a compression story

The reflex is to credit gzip, snappy, zstd — wrong layer entirely. The real story is what bytes get pulled off disk at all. A row store (CSV, JSON lines, a plain row-oriented DB page) writes each row as one contiguous chunk: trace_id, latency_ms, and raw_response sitting shoulder to shoulder on disk. Check latency_ms for one row, and the I/O layer drags the giant raw_response blob along for the ride — for every single row. You can't check one column without paying for all of them. Parquet breaks that coupling by storing columns physically apart, and once they're apart, the engine can skip whole groups of them without asking the compressor to lift a finger.

Anatomy, fast

A Parquet file is four nested layers, and the footer is the part almost nobody looks at:

• ▹File — the whole .parquet object, footer at the very end (readers seek to the end first, not the start — that's not an accident)
• ▹Row group — a horizontal slice of the table, typically ~128MB–1GB of rows, self-contained enough to read on its own
• ▹Column chunk — inside a row group, one contiguous block per column. This is the columnar part.
• ▹Page — a column chunk split into smaller units (default ~1MB), the actual unit of encoding and compression
• ▹Footer — schema, plus per-column-chunk statistics (min, max, null count, distinct estimate) for every row group in the file

The actual trick: two pushdowns, not one

The engine opens the file, reads the footer first — it's small, one seek — and runs two separate skip operations before it touches a single byte of real data:

• ▹Predicate pushdown — for `WHERE latency_ms > 5000`, the engine checks each row group's stored max(latency_ms) in the footer. If a row group's max is 3000, that whole row group gets discarded — unread, undecompressed, untouched. This is why a table sorted or clustered well (ordered by time, say) skips almost everything: the min/max ranges barely overlap.
• ▹Projection pushdown — for `SELECT trace_id, latency_ms`, the engine never opens the column chunk for raw_response in any surviving row group. You pay I/O and decode cost only for the columns you actually asked for.
• ▹These two multiply. Predicate pushdown throws out most row groups outright; projection pushdown means the survivors only get charged for the columns in the SELECT list — here, trace_id and latency_ms, never raw_response. That combination is the real source of the 44x. Decompression is what happens to the small fraction of bytes left standing after both skips — it's not the mechanism that produced the speedup.

Why same-typed columns compress absurdly well

Compression is real — it's just downstream of the layout decision, not the cause of the speed. Once a column chunk holds nothing but one type of value, three cheap encodings become viable that a row store can never touch, because a row store's disk-adjacent bytes read `int, string, float, string, int...` — no repeating structure for any of these to grab onto:

• ▹Dictionary encoding — a column like model_name with 6 distinct values (opus, sonnet, haiku...) across a billion rows becomes a small dictionary plus tiny integer references. Biggest single win for low-cardinality columns: agent names, status codes, environment tags.
• ▹Run-length encoding (RLE) — a success boolean that's true for 10,000 rows in a row collapses to one (value, count) pair instead of 10,000 repeated bytes.
• ▹Delta encoding — sorted or near-sorted numeric columns (timestamps, auto-increment IDs, token counts in an ordered log) store the difference from the previous value instead of the full number, which is usually tiny.
• ▹Mix types in one physical block, like a row store does, and none of this applies. You're back to general-purpose byte compression — which is exactly why 'just gzip the CSV' never gets within shouting distance of Parquet's ratio.

The catch: this trick has a shape it needs

This is exactly where 'just convert to Parquet' backfires — and it's the part most teams learn the hard way, often after wiring Parquet under an AI pipeline that was writing like a database underneath.

• ▹Row groups too small (a few thousand rows) — footer statistics become nearly meaningless, since every group's min/max range overlaps every other's. Predicate pushdown skips nothing, and you pay per-row-group overhead (footer entries, page headers) for zero benefit.
• ▹Thousands of small files — one Parquet file per micro-batch, per agent run, per minute is a common failure mode. Each file carries its own footer, its own open/seek/read cost, so the engine spends more time on file-open overhead than on actual scanning. This is the single most common cause of 'we moved to Parquet and it got slower.'
• ▹Single-row updates or appends — Parquet has no update-in-place. A one-row change means rewriting the row group, or the whole file. This is fundamentally an OLAP shape — write big batches, read with selective filters — not an OLTP shape of insert-one-row, read-one-row, repeat. If your access pattern is 'one log line, one write,' Parquet is the wrong tool at the write layer. Buffer and batch first — a queue, an OLTP store, Kafka — then flush to Parquet.

This is the sharp edge in agentic pipelines specifically: frameworks that log every tool call or LLM turn as its own immediate write, straight to Parquet, produce exactly the small-file, high-overhead antipattern above. Batch the writes — per session, per minute, per N events — before you ever call the Parquet writer.

Field guide: inspect your own files

Don't take the format's reputation on faith. Read the footer yourself.

# parquet-tools (or parquet-cli) — see row group count, sizes, and stats
parquet meta my_table.parquet
parquet footer my_table.parquet   # some versions expose this subcommand

import pyarrow.parquet as pq

pf = pq.ParquetFile("my_table.parquet")
print(pf.num_row_groups)

for i in range(pf.num_row_groups):
    rg = pf.metadata.row_group(i)
    col = rg.column(rg.num_columns - 1)  # pick a column index you care about
    stats = col.statistics
    print(i, rg.num_rows, stats.min, stats.max, stats.null_count)

• ▹Sane default: target row groups in the 128MB–512MB range (Spark, DuckDB, and Athena each have their own sweet spot, but this range is a safe start), and consolidate small files with a compaction job instead of writing one file per micro-batch.
• ▹Warning sign your 'fast' Parquet table is actually slow: `pf.num_row_groups` in the thousands for a table under a few GB, or `ls` showing thousands of files under 10MB each. Both mean you're paying footer/open overhead on every query with none of the skip benefit.

Bridge to Day 10

Storage skipping the right bytes only gets you to the survivors — the row groups and columns that clear both pushdowns. What the engine then does with those surviving bytes, row by row versus batch by batch, is a separate question: vectorized execution. That's Day 10.

Extend your knowledge

• ▹Run `parquet meta` (or the pyarrow snippet above) on a real Parquet table your team owns — check row group count and file count before you trust that it's 'fast.'
• ▹Read the Apache Parquet format spec's footer/FileMetaData section directly — it's short, and it's the ground truth for what stats actually get stored per column.
• ▹Compare DuckDB's `EXPLAIN ANALYZE` output on a filtered query against a Parquet table versus a CSV of the same data — the row-groups-scanned count in the plan makes the skip trick visible.
• ▹Look at how your team's agent/LLM logging pipeline writes to Parquet today — check whether it batches writes or does one-file-per-event, since that's the exact failure mode covered above.