Apache Parquet, Explained

For years, data analytics was built on formats designed for something else entirely: CSV and JSON are convenient to generate and read line by line, but a typical analytical query doesn't read lines — it reads columns. A query like SELECT avg(amount) FROM sales WHERE date = '2026-08-01' doesn't need to see 90% of the fields in each row, but a row-oriented format forces you to read them anyway. Apache Parquet was built to solve exactly that mismatch: a columnar, binary storage format with embedded metadata, designed so a query engine reads only what it needs and discards the rest without even opening it. That property — combined with aggressive compression and cross-engine compatibility — is why it's now the default format in any serious data lake or lakehouse, from Spark to Trino, Athena, or DuckDB.

The anatomy of a Parquet file

A Parquet file isn't a flat blob — it has an internal hierarchy built to allow selective reading at different levels of granularity. At the top level, a file is split into row groups, horizontal slices of the dataset (roughly 128 MB by default). Within each row group, data is reorganized by column into column chunks: all values of a given column, for that subset of rows, sit contiguously on disk. And within each column chunk, values are further split into pages (around 1 MB by default), which are the minimum unit of reading, decoding, and compression.

Each page carries, alongside the encoded values, the repetition and definition levels Parquet uses to represent nested and optional fields without having to flatten the structure. The whole file closes with a footer that holds the full schema, the format version, and statistics (min, max, null count) per column chunk. That footer is what makes a Parquet file self-describing: any engine can read it without an external catalog to know which columns it has and what range of values each block contains.

Why the columnar model enables compression and pushdown

Grouping values from the same column, of the same type, has two direct consequences. The first is that compression works better: a column of statuses ("active", "active", "inactive"...) or of dates has far less entropy than a full row mixing types, so techniques like run-length encoding or dictionary encoding shrink the data aggressively even before a generic compressor like Snappy, Gzip, or ZSTD is applied. The second consequence is that the engine can decide, column by column, whether it needs to read it at all. If your query only touches three columns out of forty, the rest of the column chunks never even get decompressed.

On top of that foundation sits predicate pushdown: since the footer stores the min and max of each column chunk, the engine can compare the query's predicate against those ranges and skip entire row groups without reading a single byte of data. If you filter on x > 5 and a row group has x ranging from 0 to 4, that block gets discarded before it's ever opened. When a column uses dictionary encoding, filtering can be even more precise: if the value you're looking for doesn't even appear in that page's dictionary, it's discarded without comparing row by row.

This optimization has limits worth knowing. It doesn't work well on unsorted data, because if a column's values are scattered randomly across row groups, almost no min/max range lets you discard anything — for pushdown to be effective, data has to be written sorted (or clustered) by the columns that get filtered most. It's also not equally effective across data types: comparisons on integers, strings (equality only), or booleans are straightforward, but on floats or decimals the precision of the binary representation can introduce inaccuracies. And you need to use predicates of the same type as the column — comparing a long column against an int literal can prevent the engine from applying pushdown at all.

Parquet doesn't speed up queries by itself: it speeds up queries that are written and organized to avoid reading what isn't needed.

Complementing statistics-based pushdown is partition pruning: organizing data into subdirectories of the form column=value so the engine can discard entire partitions before even listing files. It's powerful, but it comes with tradeoffs: use low- or medium-cardinality columns (date, region, tenant), avoid partitions smaller than 1 GB, and avoid nesting too many partition columns, because each additional combination multiplies subdirectories and eventually produces the classic "small files problem."

Schema evolution without rewriting history

Since every file carries its own schema in the footer, Parquet handles a table's schema changing over time reasonably well: new columns can be added, and older files that don't have them simply return null for those fields when read; columns can be reordered without breaking reads, because mapping is done by name, not position; and nested types (structs, arrays, maps) are represented natively thanks to the repetition/definition level model, with no need to flatten the structure or serialize a JSON blob inside a column.

This isn't magic without rules: the query engine has to be configured to merge schemas across files with different versions (many frameworks leave this off by default because of listing cost), and changing the type of an existing column — from int to string, for example — remains a delicate operation that can break older readers. Schema evolution in Parquet cheaply handles the common case (adding/removing/reordering columns); type changes still require a deliberate migration.

The foundation of lakehouses: Delta Lake, Iceberg, and friends

Parquet solves the file format, but it doesn't solve the table. A directory full of Parquet files has no atomic transactions, doesn't coordinate concurrent writers, offers no time travel, and, with every micro-batch of an incremental pipeline, tends to accumulate small files that degrade read performance over time. That's where table formats come in: Delta Lake, Apache Iceberg, and Apache Hudi add a metadata layer and a transaction log on top of plain Parquet files, providing ACID transactions, versioning, file-level skipping statistics, and maintenance operations like automatic compaction (OPTIMIZE) or column clustering (Z-order).

The key is understanding the division of responsibilities: the data on disk is still Parquet — column chunks, pages, min/max per chunk — and what the table format adds is a transactional catalog that knows which Parquet files make up each version of the table and how to swap them out atomically. Parquet doesn't compete with Delta Lake or Iceberg; it's the physical layer they're built on.

When to use it in practice

• Row group size: for typical workloads in Spark or other distributed engines, aim for row groups between 128 MB and 1 GB. Row groups that are too small multiply per-file metadata overhead; too large reduces parallelism across tasks.
• Partition columns: choose low- or medium-cardinality columns that show up in your most frequent WHERE clauses (date, region, tenant). Avoid partitioning by near-unique columns (IDs, second-precision timestamps), and avoid nesting more than one or two partition columns.
• Target file size: avoid files below ~50-100 MB (the overhead of listing and parsing metadata for thousands of small files can mean a query spends more time reading footers than data), and avoid giant files of hundreds of gigabytes in a single partition, which kill parallelism and slow down even a simple count(*).
• Sort before writing: if you know which columns will be filtered most, sort or cluster by them at write time. Without that, the footer's min/max statistics are useless for discarding row groups.
• Compact regularly: in incremental or streaming pipelines, schedule compaction (manual or automatic via your table format) to avoid the buildup of small files.
• When NOT to use it: for transactional workloads with row-by-row writes and updates (OLTP), for random access to individual records, or for small datasets where columnar metadata overhead doesn't pay for itself. In those cases, a relational database or a row-oriented store is still the right call; and if you're operating on a lakehouse anyway, it's better not to write plain Parquet but to wrap it in Delta Lake or Iceberg from day one.