Monzo's Governed Data Mesh Architecture for Scalable Data Warehousing
Monzo, a neobank, redesigned its data warehouse into a governed data mesh to manage over 12,000 dbt models across 100 teams. This architectural shift improved data delivery speed by 25% and reduced warehouse costs by 40% through formalizing data sharing, enforcing standards with CI, and automating data quality checks. The system supports distributed ownership of data models while ensuring consistency and performance at scale.
Read original on InfoQ ArchitectureIntroduction to Monzo's Data Mesh Challenge
Monzo faced significant challenges scaling its data warehouse to support over 100 independent teams, each contributing to more than 12,000 dbt (data build tool) models. The primary issues stemmed from distributed ownership leading to inconsistencies, redundant queries, recomputation, and escalating warehouse costs. To address this, Monzo adopted a "meshy" approach, transforming its data platform into a governed data mesh.
Core Principles of Monzo's Data Architecture
- Enforce Clear Standards: Establish explicit rules for data modeling, naming conventions, and metadata.
- Formalize Data Sharing: Define explicit interface models for cross-team data dependencies.
- Rely on Automation and CI Checks: Implement automated guardrails and continuous integration (CI) for validation of structure, naming, access patterns, and quality, minimizing manual review.
Layered Data Model Structure
Monzo's data models are structured into four distinct layers, each serving a specific purpose in the data transformation pipeline:
- Automated Landing Models: Raw events are flattened into initial models.
- Generated Normalized Models: These represent entities with their full historical context.
- Logical Models: Business logic is applied here, combining various entities.
- Presentation Models: Tailored datasets for specific downstream applications and user consumption.
Governance and Automation Mechanisms
To maintain consistency and quality across thousands of models and hundreds of teams, Monzo implemented several governance and automation mechanisms:
- Modelgen: A command-line tool that generates SQL and YAML models from object definitions, ensuring standardized creation.
- CI-backed Data Standards: Strict validation of structure, conventions, and best practices integrated into the CI pipeline.
- Data Quality Enforcement: Each model is required to define a unique key, include freshness tests, run incrementally by default, declare an owning team, provide documentation, and adhere to strict naming and metadata conventions.
Architectural Benefits
This data mesh approach enabled Monzo to achieve a ~40% cost reduction in warehouse expenses and ~25% faster data landing times in some domains. It highlights how strong governance, automation, and clear architectural boundaries can significantly improve efficiency and scalability in complex data environments, even with distributed ownership.