Netflix's DataJunction: A Semantic Layer for Consistent Metrics and Experimentation
Netflix implemented DataJunction, an open-source semantic layer, to address challenges with scattered, inconsistent metric definitions within its experimentation platform and broader analytics. DataJunction streamlines data workflows by centralizing metric and dimension definitions in a connected graph, enabling consistent SQL generation, and improving data discoverability and trust. This architectural decision enhances data-driven decision-making and accelerates metric onboarding for data scientists.
Read original on Netflix Tech BlogThe Challenge of Inconsistent Metrics at Scale
At large, data-driven organizations like Netflix, a common problem is the proliferation of inconsistent and scattered metric definitions. This leads to redundant datasets, complex pipelines, and a lack of trust in data, particularly within critical areas like experimentation platforms where reliable metrics are paramount for interpreting A/B test results. The existing system at Netflix's Experimentation Platform required extensive business knowledge and complex engineering practices, leading to metric authoring processes that could take weeks, significantly hindering data scientists' ability to onboard new metrics efficiently.
DataJunction as a Semantic Layer Solution
To overcome these hurdles, Netflix adopted DataJunction (DJ), an open-source semantic layer solution. Key architectural differentiators that led to its selection included its robust SQL parsing and SQL generation engine, and its ability to store all metric and dimension definitions in a connected graph. This graph-based approach, similar to how relational databases manage view metadata, facilitates "SQL wrangling" with rich metadata, offering metrics portability and enabling seamless migrations by abstracting metric definitions from the underlying computation platform.
Core Problem Addressed by DataJunction
DataJunction tackles the critical system design problem of data discoverability, consistency, and governance for metrics in complex, distributed data environments. It centralizes definitions to prevent metric sprawl and ensure a single source of truth for analytical insights.
Architectural Decisions and Focus
Given resource constraints, the development of DataJunction at Netflix prioritized an API-first approach, focusing on delivering a robust semantic layer service rather than a complex UI. The core system design emphasized a "Consistent Graph First" strategy, building out the solid semantic layer engine capable of parsing SQL input into its graph representation and generating SQL for serving. This foundation allows for incremental addition of other features like data visualization, caching, and integrations, without rebuilding the core.
DataJunction adopted analytical cubes as a primary consumption protocol. Cubes serve as a familiar and efficient interface for delivering both semantic definitions (metrics and dimensions) and fast data access. This design choice simplifies integration with existing tools like Netflix's Druid data analysis tool and external platforms like Superset, enabling easy linking of dashboard metrics back to their DJ-defined source for introspection and troubleshooting.
- API-first design: Semantic layer served via a rich API, allowing various clients to consume definitions and data.
- Graph-based metadata: Centralizes metric and dimension definitions in a connected graph for consistency and discoverability.
- SQL parsing and generation: Core engine for converting user definitions into executable SQL against underlying data sources.
- Cubes as consumption protocol: Simplifies access to aggregated metrics and dimensions for data analysis tools.
- Integration layer: Designed for easy integration with internal and external data visualization and analysis platforms.