Apache Iceberg: Open Tables Against Engine Lock-In

Any team that has run a data lake for more than two years knows the pattern: you pick a query engine, build the entire pipeline around its quirks, and two years later migrating to a different engine — or simply adding a new one for a different use case — means rewriting the whole storage layer. The problem isn't the engine itself. It's that the format the data lives in ended up coupled to it. Apache Hive solved the original data lake problem (running SQL over flat files) but did so by tying the physical layout of the data — partition folders, directory listings to discover files — to a metastore that every engine had to interpret in its own way. That implicit dependency between engine and format is exactly what open table formats like Apache Iceberg were built to break. The promise isn't "a better engine" — it's that the table format stops being an infrastructure decision that locks you into a vendor.

The layered architecture: catalog, metadata, manifests, and data

Iceberg doesn't store a table's definition as a database entry pointing at a folder. It builds it as a chain of immutable files, each with a single, concrete responsibility. A catalog keeps, for each table, a pointer to the current metadata file. That metadata file (JSON) describes the schema, the partition scheme, and the table's list of historical snapshots. Each snapshot references a manifest list (in Avro), which in turn enumerates one or more manifest files, and those list the actual data files (typically Parquet) along with per-column statistics: value ranges, counts, nulls.

This chain matters because it completely changes how an engine "finds" data. In Hive, knowing which files belong to a table requires listing the underlying storage system — an expensive operation, and on object stores like S3, one with consistency guarantees that get unpredictable under concurrent writes. In Iceberg, the engine never lists directories: it walks metadata that already knows the exact location of every file, and thanks to the embedded statistics, it can discard entire files before reading a single byte of data. The practical result is twofold: cheaper queries (less I/O) and, more importantly, real ACID transactions. A write is only considered "committed" once the catalog pointer is atomically updated to point to the new metadata file. If the process fails halfway through, readers keep seeing the previous snapshot, intact. That's something Hive, by design, cannot guarantee.

Hidden partitioning and partition evolution

In a Hive-style data lake, the partition scheme is literally the folder structure: if the table is partitioned by date, users need to know there's a physical column with that name and filter explicitly on it for the engine to take advantage of partitioning. If someone writes WHERE event_timestamp BETWEEN ... instead of filtering on the derived partition column, the engine ends up scanning the entire table without anyone noticing until the compute bill arrives.

Iceberg separates the column the user queries from the transformation that defines the partition. You can partition by day(event_timestamp) or month(creation_date), and the engine applies that transformation automatically whenever it detects a filter on the original column, without the user needing to know or reference the partitioning detail at all. This is called hidden partitioning, and it eliminates an entire class of silent performance bugs.

The second piece is partition evolution: you can change a table's partition scheme — say, moving from monthly to daily partitioning — without rewriting a single existing data file. Old snapshots keep their original partition scheme; new ones use the updated scheme. Both coexist in the same table because the partition scheme lives in each snapshot's metadata, not in the physical folder structure.

Partitioning stops being a decision made once, at table creation, and becomes a parameter you tune as volume or query patterns change.

Schema evolution without rewrites

The same principle — metadata as the source of truth, physical data as a low-level implementation detail — applies to the column schema. Iceberg identifies each field by a unique internal ID, not by name or position. That makes it possible to rename columns, add new columns, drop columns, or even reorder them without touching the underlying Parquet files: the engine resolves the mapping between logical ID and physical position at read time. This evolution extends to nested fields in complex structures too, something that in more rigid formats usually forces a full table rewrite.

This same metadata-versioning capability enables time travel: since every transaction generates a new snapshot without destroying the previous one, you can query a table's exact state at any point in its history, or even create branches and tags on specific snapshots with their own retention policies. It's a deliberately git-like approach applied to data, useful both for auditing and for recovering from a bad load without restoring backups.

Engine-agnostic catalog: the real defense against lock-in

Everything above would be an interesting but secondary format improvement if it weren't paired with a deeper design decision: Iceberg is a specification, not an implementation tied to a runtime. Spark, Trino, Flink, Snowflake, DuckDB, or Dremio can all read and write the same Iceberg table with no prior coordination between them, because they all implement the same table format specification. This contrasts with formats originally designed around one specific engine and only later gaining third-party support — there, interoperability is an afterthought, not part of the original design.

The historical friction point was the catalog: every catalog implementation (Hive Metastore, Glue, Nessie, proprietary catalogs) required each engine to write a separate client in each language. That led right back to the same coupling problem, just one level up. The REST Catalog Specification solves this by standardizing the communication protocol between engine and catalog: any engine that speaks Iceberg's REST protocol can operate against any catalog that exposes it, regardless of what language either side is written in.

When to use it in practice

• If you already run more than one query engine in production (say, Spark for batch and Trino or Snowflake for BI), Iceberg avoids keeping duplicate copies of the same data in different formats for each engine.
• If your tables receive frequent concurrent writes — CDC, parallel backfills alongside incremental ingestion — Iceberg's ACID guarantees eliminate a whole class of duplicate or inconsistent data bugs that in Hive-style setups require manual coordination between jobs.
• If your table schemas change regularly (new columns, renames, type changes), rewrite-free schema evolution saves hours of maintenance every time it happens.
• If you suspect the query engine you use today won't be the same one two or three years from now — due to cost, capacity, or organizational decisions — building on Iceberg with a REST catalog is how you avoid paying for that migration by rewriting terabytes of data.
• Don't introduce it if your data volume is small and a single engine covers all your needs indefinitely: the metadata layer adds operational complexity (snapshot compaction, manifest file management) that's only worth it when volume or engine heterogeneity demands it.
• Avoid the filesystem-based catalog (version-hint.text) in production: it doesn't offer consistent atomic-write guarantees across all storage backends. Always use a service-backed catalog.