Delta Lake: ACID on Top of Your Data Lake

A data lake built on plain Parquet in S3, ADLS, or GCS is, at bottom, a collection of immutable files with no arbiter deciding which version of the truth is correct. If two jobs write to the same table at the same time, there's no locking and no isolation: whoever finishes last wins, and if a job fails mid-write, partial files are left behind that a reader can pick up as if they were valid data. There's no real row-level UPDATE or DELETE: the usual practice is to rewrite entire partitions. And if you need to know what the table looked like yesterday at 3pm to debug a pipeline or audit a change, there's nowhere to look. This isn't a flaw in Parquet as a file format — it's a consequence of having no transactional layer on top of it. Delta Lake attacks exactly this problem: it adds metadata and a commit protocol on top of the same Parquet files to turn a directory into something that behaves like a table.

The transaction log as the single source of truth

The central piece of Delta Lake is a subdirectory called _delta_log that lives alongside the data files. Every operation on the table (an INSERT, a MERGE, a schema change) produces a sequentially numbered entry in that log, in JSON format: 00000000000000000000.json, 00000000000000000001.json, and so on. Each entry doesn't contain the data itself, but actions: which Parquet files were added, which became obsolete, whether the schema changed, transaction metadata. A reader never "guesses" the state of the table by scanning the data directory; it reconstructs that state by replaying the log in order, the same way a database engine replays its write-ahead log.

This is deliberately similar to how Git resolves a branch's history: a sequence of commits that can be replayed forward or queried at any intermediate point. Because that reconstruction would be expensive if it meant reading thousands of JSON files, Delta Lake automatically writes a checkpoint in Parquet every ten commits, with the consolidated state up to that point. A new reader only needs the latest checkpoint plus the commits after it, not the full history since day one.

One detail that surprises people coming from traditional databases: deleting or overwriting a row doesn't physically delete the Parquet file that contained it. Delta Lake marks that file as "removed" in the log and adds a new one with the updated data. The old file stays on disk until VACUUM runs, which is the explicit command to purge files no longer referenced by any version within the retention window.

What ACID actually guarantees here

On object storage, which has no native locks or transactions, ACID is implemented at the metadata layer, not in the underlying storage. Atomicity means a transaction translates into a single new entry in the log: if the process writing the data fails before that entry is committed, the table simply saw no changes, with no orphaned files contaminating future reads. Consistency is enforced through schema enforcement on every write: if the incoming schema doesn't match the table's (incompatible types, undeclared new columns, differences in column name capitalization), the transaction is rejected before it ever touches disk.

Isolation is handled through optimistic concurrency control: before confirming its commit, each writer checks whether someone else has already written a newer version that its transaction would conflict with; if so, it retries against the new version instead of corrupting the result. Durability is the most straightforward property: once the commit's JSON file is confirmed in the underlying object storage, the transaction is final and visible to any reader that queries the log from that point on.

The transaction log isn't an implementation detail: it's literally the mechanism that makes "reading a Delta table" an operation with a single, consistent answer, instead of a race between writers.

Time travel: versioning without manual backups

Since every version of the table is recorded as a reproducible sequence of commits, querying historical state doesn't require having taken a manual snapshot beforehand. You can read a table as it stood at a numbered version or at a specific timestamp, and you can revert the entire table to a prior state with RESTORE. This natively solves several problems that previously required separate infrastructure: regulatory audits of what the data looked like on a given date, reproducibility of a machine learning experiment trained against a specific version of the dataset, or simply undoing an UPDATE without a WHERE clause that you just ran by mistake.

This capability comes with a storage cost that has to be actively managed. Data files don't delete themselves; they stick around as long as some retained version references them. The default data retention window is 7 days, configurable per table, and the retention for the _delta_log log files themselves is 30 days. If you need time travel a year back for a regulatory requirement, both retention windows have to be extended explicitly, and you have to accept that storage will grow proportionally. For very long retention horizons, it's usually cheaper to archive periodic snapshots into separate tables than to keep the full history alive in the operational table.

Real upserts and controlled schema evolution

Unlike plain Parquet, where an "upsert" means rewriting the entire partition or juggling external logic, Delta Lake supports full DML: UPDATE, DELETE, and above all MERGE INTO to combine an incoming batch with an existing table in a single atomic operation. Under the hood this still operates at the file level (the affected Parquet files get rewritten, not individual rows), but from the perspective of whoever is writing the pipeline it's a declarative statement, not a handcrafted process.

Schema management has two sides that are worth keeping separate. Schema enforcement is the guardrail that keeps incorrectly shaped data out of the table: by default, a write with undeclared new columns fails. Schema evolution is the opposite, complementary mechanism: when explicitly enabled with the mergeSchema option, a new column in the incoming data gets added to the table's schema, backfilling historical rows with null. It's additive, controlled evolution, not an "anything goes" policy: incompatible type changes still fail, and renaming or dropping columns requires explicit ALTER TABLE statements — it doesn't happen by accident on a normal write.

Performance: compaction and Z-order

The same mechanisms that provide reliability have a side effect: frequent incremental writes (streaming, micro-batches) generate lots of small Parquet files, which hurts read performance because every open file carries fixed overhead. The OPTIMIZE command compacts those small files into well-sized ones without changing the table's logical content, and it's idempotent: running it twice in a row does no harm and doesn't duplicate work.

Delta Lake also keeps per-file statistics (minimums, maximums, null counts) on the first columns of the schema, which lets it skip entire files at query time when a filter falls outside their range: this is data skipping, and it's why ordering the data matters. ZORDER BY, combined with OPTIMIZE, physically reorganizes the data so that similar values in the specified columns end up placed together, narrowing those ranges and making data skipping much more effective. The practical limitation is that Z-order isn't incremental: every run reclusters the entire affected dataset, so on tables with continuous writes it's worth scheduling it at a reasonable cadence instead of running it after every micro-batch. Liquid clustering is the more recent evolution of this idea: it replaces both classic partitioning and Z-order with a mechanism that does support incremental clustering on every write, aimed at tables with high filter cardinality or access patterns that shift over time.

When to use it in practice

Delta Lake makes sense almost by default in any lakehouse where multiple jobs write to or read the same table concurrently, where batch and streaming processes coexist on the same source, or where the pipeline needs real upserts instead of full rewrites. It's also the right choice when there's an audit or regulatory compliance requirement around historical data, or when data quality is a recurring problem and you want a write with an incorrect schema to fail loudly instead of slipping through silently.

Don't adopt it just because "it's the modern thing to do." If your workload is predominantly read-only, with data written once and never changed, and you don't need versioning or write concurrency, the overhead of maintaining _delta_log, periodically running OPTIMIZE and VACUUM, and managing retention windows is cost without a matching benefit. It also doesn't make sense if your query stack doesn't have a reasonable Delta connector; although the ecosystem has grown (Spark, Trino, Presto, Flink, Power BI via Delta Sharing, and UniForm for interoperating with Iceberg and Hudi), it's still worth checking support in your specific engine before committing the architecture to it.