CSV, JSON, Avro, Parquet: How to Choose a File Format

Almost nobody argues about file format during the design phase of a pipeline. CSV gets picked because "that's how the data arrives," or JSON because "that's what the API returns," and the topic gets closed. The problem is that this decision, made without much thought, propagates: it determines how much space the data takes up in the lake, how long a Spark job takes to scan a table, how much each query costs in an engine that bills by bytes read, and how painful it is to add a new column without breaking existing consumers. File format isn't an implementation detail — it's an architectural decision with direct impact on cost and performance, and like any architectural decision, it deserves to be made on purpose.

Row-oriented vs. columnar: the decision that explains almost everything

The underlying question behind any analytical storage format is a single one: how are the bytes physically laid out on disk? There are two classic answers.

In a row-oriented (row-wise) format, each record is written in full before moving to the next. All the values of row 1 sit contiguously, then all of row 2, and so on. This is excellent when the workload is transactional (OLTP): you need to read or write an entire record — an order, a user, a transaction — and want to do it in a single I/O operation.

In a columnar format, by contrast, values from the same column are grouped together on disk: all the "customer_id" values contiguous, then all the "email" values, then all the "create_date" values. This favors analytical workloads (OLAP), where a typical query aggregates or filters over a handful of columns but scans millions of rows. If all you need is the average of one numeric column across a hundred-column table, a columnar engine reads only that column from disk; a row-oriented engine has to pull entire records and discard 99% of the bytes it read.

The real cost of an analytical query is almost never the CPU: it's how many bytes you had to read from disk to answer it.

Columnar organization also enables compression techniques that don't work nearly as well in row-based formats, such as run-length encoding or dictionaries of repeated values, because a single column tends to have far less value variability than a full row. There's also a third, hybrid approach (sometimes called PAX), used by formats like Parquet and ORC, which groups rows into blocks and organizes the data by column within each block — giving you compression and columnar pruning without fully losing row locality.

CSV and JSON: still the right choice in plenty of cases

It's tempting to treat CSV and JSON as "legacy" formats to be replaced as soon as possible, but that ignores what they were designed for. They're textual formats, human-readable, editable in any text editor, and inspectable without depending on a specific library. That portability has real value.

CSV remains the lingua franca for exchanging simple tabular data: exports from legacy systems, manual uploads, integrations with business tools that don't speak anything else. Its structural limitation is that it carries no types or metadata: everything is plain text, so whoever consumes it has to infer or cast types, deal with inconsistent encodings, and trust that the delimiter never shows up inside a value.

JSON starts to make sense as soon as the data stops being flat. It's semi-structured: it supports nested objects, arrays, and records of the same "entity" with different fields from one another, without breaking the file. It's the natural format for REST API payloads, application logs, tracking events, or documents with variable structure. The cost is the mirror image of CSV's: the flexibility means there's no schema guarantee, every parser has to validate structure at read time, and the footprint on disk is larger because field names repeat in every record.

The practical rule: if the data is small, shared between humans and heterogeneous systems, or its structure changes from one record to the next, CSV or JSON are still reasonable choices. The problem shows up when they're used as the underlying storage format for large datasets that get queried repeatedly for analytics.

Avro: when the schema evolves and the data travels row by row

Avro is a row-oriented binary format, but with one crucial difference from CSV: the schema is part of the contract. Every Avro file embeds (or references, via a schema registry) a formal JSON definition of its fields, types, and default values, which lets producers and consumers evolve independently without one breaking the other. You can add an optional field, retire one with a default value, or rename it with an alias, and old and new readers stay compatible.

That schema-evolution capability, combined with its row-wise nature, makes it the natural choice for streaming and messaging systems: writing a full record is a cheap, local operation — you don't have to touch multiple scattered columns on disk. It's the default format in ecosystems like Kafka, where events are written one at a time at high frequency, and where the payload schema can change over time without coordinating a simultaneous deployment across every producer and consumer.

The trade-off is predictable: being row-wise, analytical queries that only need a subset of columns don't benefit from columnar pruning, and compression is less aggressive than in a columnar format because values from the same column aren't stored contiguously.

Parquet: the default format for analytics at scale

Parquet is columnar (in practice, a PAX-style hybrid) and is today the de facto standard for storing data in a lake or lakehouse that will be queried with engines like Spark, Trino, Presto, or Athena. Three properties explain why it dominates the analytical space.

First, columnar pruning: if a query only touches 3 of 50 columns, the engine physically reads only the blocks for those 3, which directly cuts I/O and, in serverless engines that bill by bytes scanned, cuts query cost too. Second, compression: homogeneous values grouped by column compress far better than heterogeneous rows, translating into a smaller storage footprint. Third, splittability: a Parquet file is organized into independent row groups with their own metadata and statistics (min, max, null counts), which lets a distributed engine parallelize reads across multiple workers and, on top of that, skip entire row groups when it already knows they don't contain the values a query is looking for.

The cost of this read efficiency is that writes are heavier: you can't simply append a row to the end of the file the way you can with CSV — entire blocks have to be rewritten — which is why Parquet isn't the right choice for transactional workloads or high-frequency row-by-row writes. Its schema-evolution support is also more limited than Avro's: adding columns usually works fine, but type changes or reordering require more care.

When to use it in practice

• If the end consumer is human, or the data is shared between disparate tools with no dedicated libraries for reading binary formats: use CSV. Accept the cost of having to cast types and validate the delimiter.
• If the data has nested or variable structure, and the consumer is a service or API that already works natively in JSON: use JSON, or its NDJSON variant (one object per line) if you need splittability for parallel processing.
• If you're building a streaming pipeline (Kafka, Kinesis) or any system where the payload schema will change over time and producers and consumers can't be allowed to break each other: use Avro, backed by a schema registry.
• If the data's final destination is analytics — aggregate queries, dashboards, engines like Spark, Trino, or Athena over a data lake — convert to Parquet in the curation layer, even if the data arrived in a different format. The savings in query cost and scan time almost always justify the conversion step.
• If the dataset needs upserts, deletes, or version control on top of Parquet or ORC files — not just the file format but how the table itself is managed — consider an open table format such as Delta Lake, Apache Iceberg, or Apache Hudi on top of Parquet, since they add ACID transactions, time travel, and schema and partition evolution without requiring you to rewrite the entire pipeline.
• Avoid the most common mistake: leaving high-volume, frequently-queried data in JSON or CSV just because "that's how it arrived." That's exactly the scenario where the cost of not converting to a columnar format gets paid, with interest, on every single query.