Indexing Data Lakes for Low-Latency Online Point Queries
This article discusses Spotify's approach to making vast quantities of data stored in data lakes accessible for online services with low latency. It focuses on the challenges of efficiently indexing and querying historical data, often stored in formats not optimized for point lookups, and the architectural considerations for building a system that can serve these queries in real-time.
Read original on Spotify EngineeringCompanies like Spotify accumulate massive amounts of data in data lakes, typically stored in cost-effective, but slow-to-query, formats like Parquet or ORC on object storage (e.g., S3). While excellent for analytical workloads and batch processing, these formats and storage solutions are not designed for the low-latency online point queries required by many user-facing services or internal applications. The core challenge is bridging the gap between batch-optimized storage and real-time query needs.
The Challenge: Data Lakes vs. Online Queries
Data lakes excel at storing raw, diverse data at scale and are ideal for retrospective analysis, machine learning model training, and long-term archival. However, when an online service needs to retrieve a specific record (e.g., a user's listening history for a recommendation, or account details for a profile page) with millisecond latency, directly querying a data lake is impractical due to high read latencies and the full-scan nature of most data lake query engines for non-indexed data. This necessitates an indexing layer that can efficiently serve point lookups over historical data.
Architectural Considerations for Online Indexing
- Data Ingestion Pipeline: How is data efficiently moved from the data lake (or streaming sources) into an indexed store? This often involves stream processing (e.g., Kafka, Flink) to transform and load data.
- Indexing Strategy: What kind of index is needed? Is it a simple key-value store, a column-family store, or something more complex? The choice impacts query flexibility, storage cost, and write performance.
- Consistency Models: What level of data freshness and consistency is required? Eventually consistent might be acceptable for some use cases, while strong consistency is critical for others.
- Scalability and High Availability: The indexing solution must scale horizontally to handle growing data volumes and query loads, and be resilient to failures.
- Cost-Effectiveness: Balancing the need for low latency with operational costs (storage, compute, network) is crucial. Indexing everything for all possible queries can be prohibitively expensive.
Key Trade-off: Query Flexibility vs. Latency/Cost
Designing an online indexing solution involves a fundamental trade-off. Highly optimized, specialized indices (like a simple key-value store) offer the lowest latency for specific query patterns but sacrifice flexibility for ad-hoc queries. More general-purpose indices or analytical databases can offer more flexibility but might come with higher latency or operational costs for point lookups.