Architecting Data Lakes with Interoperable Table Formats: Delta Lake, Iceberg, and UniForm

The debate between Delta Lake and Apache Iceberg often stems from a fear of vendor lock-in, overlooking the increasing interoperability between these table formats. Modern data architectures are moving towards abstracting the underlying storage format, treating it as an implementation detail rather than a core architectural decision. This shift is driven by solutions that allow different query engines and data processing tools to access data seamlessly, regardless of the original table format.

The Rise of Interoperability Layers (e.g., Databricks UniForm)

Tools like Databricks UniForm act as translation layers, enabling a Delta Lake table to be simultaneously exposed as an Iceberg-compatible table. When UniForm is enabled, an asynchronous background process generates Iceberg metadata (like `metadata.json` and manifest files) alongside the Delta transaction logs. This allows Iceberg-native engines (e.g., Trino, StarRocks) to read the same underlying Parquet files as if they were native Iceberg data, while Delta's high-performance write features (deletion vectors, Z-Ordering) are maintained.

ALTER TABLE my_table SET TBLPROPERTIES ('delta.universalFormat.enabledIceberg' = 'true');

Architectural Trade-offs of Cross-Format Solutions

While offering significant benefits, adopting a cross-format architecture introduces several trade-offs that system architects must consider:

• Metadata Bloat and Write Amplification: Doubling the metadata overhead can occur, leading to increased storage costs and potential lag in asynchronous translation, causing downstream readers to miss recent data.
• Dual-Writer Trap and Consistency Issues: Concurrently writing to both Delta and Iceberg metadata can lead to `ConcurrentModificationException` errors, which are difficult to debug.
• Schema Evolution Constraints: Bridging formats forces adherence to the stricter intersection of their schema evolution rules, potentially breaking complex `ALTER TABLE` operations that one format supports but the other doesn't.
• Maintenance Overhead: Managing two sets of vacuuming and snapshot expiration policies introduces complexity and can result in orphan metadata files if not handled properly.