Architecting a Governed Lakehouse Across Fabric and Azure Databricks with Zero-Copy Integration
This article details architectural patterns for integrating Microsoft Fabric and Azure Databricks into a unified, governed lakehouse, moving beyond traditional data duplication. It highlights how shared foundations like OneLake and Unity Catalog enable zero-copy data sharing and consistent governance policies across both platforms. The focus is on strategic choices for write path ownership and leveraging new integration mechanisms to build a robust, interoperable data architecture.
Read original on Dev.to #architectureHistorically, integrating Azure Databricks and Microsoft Fabric (or Synapse) meant maintaining separate data copies, distinct governance models, and complex ETL jobs. This article explains how recent advancements, particularly the convergence on OneLake as a shared physical foundation and Unity Catalog for governance, now enable a unified, zero-copy lakehouse architecture.
Key Integration Mechanisms for Zero-Copy Architectures
The article outlines three primary mechanisms for integrating Fabric and Databricks, each with different data-flow semantics and implications for architectural decisions:
- Mirrored Azure Databricks Catalog: Fabric reads Unity Catalog tables (physically in ADLS Gen2) via OneLake shortcuts. This is a read-only integration for Fabric, ideal when Databricks owns the write path. Metadata is polled and reflected.
- OneLake External Locations for Unity Catalog (Beta): Databricks writes directly to OneLake paths. OneLake becomes the primary storage, but tables remain fully managed by Unity Catalog, applying its governance and optimizations. This provides a single physical copy in the Fabric-native lake.
- Publish to Fabric (Preview): A Databricks workflow to create mirrored catalog items in Fabric workspaces, regardless of the underlying storage (OneLake or ADLS Gen2). This simplifies making Databricks tables visible to Fabric users from within the Databricks console.
Architectural Decision: Who Owns the Write Path?
The most critical architectural decision is determining where managed tables physically reside and, by extension, which platform owns the write path. This choice should be driven by the core engineering processes, not vendor preference:
Strategic Write Path Ownership
If Databricks is the engineering platform (Lakeflow, DLT, Spark streaming, MLflow): Unity Catalog should remain the source of truth. Options include keeping tables in ADLS Gen2 with Mirrored Catalog or moving storage to OneLake via external locations. The latter is a strong long-term bet for Fabric as a first-class consumption tier, enabling direct reading without indirection, but involves using a Beta feature. If Fabric owns significant write workloads (Dataflows Gen2, Fabric Warehouse T-SQL): OneLake is the center of gravity. Databricks can then federate into OneLake via Unity Catalog federation, allowing Spark and ML workloads to access OneLake-resident data with native governance without parallel copies. The crucial takeaway is to stop treating this as a data movement problem. Copying data leads to duplicate storage costs, increased reconciliation effort, and fractured governance.
Unified Governance is the Real Integration
True integration extends beyond zero-copy data sharing to a single, consistent access policy enforced across both engines. The article highlights how OneLake security now covers mirrored Azure Databricks data, using Microsoft Entra ID groups to sync privileges. This ensures that access policies (e.g., `SELECT` on a catalog) are defined once in Unity Catalog and automatically enforced when data is queried through Fabric. This single-identity, single-policy approach is crucial for auditability and simplifies security management, extending to table, row, and column-level security across all engines, and even to AI workloads via the Unity AI Gateway.