Spotify's Random Access Parquet (RAP): Low-Latency Queries on Data Lakes

The Challenge: Data Lakes vs. Point Queries

Modern data lakes, often built on cloud object storage like Google Cloud Storage, excel at analytical processing and machine learning workloads due to their scalability and cost-effectiveness. However, they are fundamentally optimized for large-scale scans rather than individual record lookups (point queries). Traditional distributed query engines like Trino or BigQuery incur significant overhead for key-based lookups, involving query planning, metadata traversal, and extensive file discovery. Replicating petabytes of data from the data lake into operational databases (like Bigtable) for low-latency access is prohibitively expensive and creates data silos.

Random Access Parquet (RAP) Architecture

Spotify's Random Access Parquet (RAP) addresses this by introducing an external indexing layer over immutable Apache Parquet files stored in the data lake. Instead of scanning thousands of files for a specific record, RAP uses this index to map lookup keys (e.g., user IDs) directly to the precise Parquet file and row location. This allows for targeted ranged reads against object storage, drastically reducing the data scanned and improving latency.

Index Management and Data Layout Optimizations

• Append-Only Index Fragments: As new data is written into Apache Iceberg tables, an index builder generates append-only index fragments. This preserves the immutability of the underlying Parquet files, crucial for data lake integrity and analytical workflows.
• Sorted Data Layout: Sorting data by lookup key reduces the number of files accessed for a query. This is a common optimization for improving locality.
• Grouped Records: Grouping related records together ensures that accessing one record often brings related data into cache, reducing subsequent I/O.
• Interleaved Value Columns: This technique stores values from multiple columns contiguously, allowing several attributes to be retrieved in a single ranged read, which is efficient for queries needing multiple fields.
• Covering Indexes: Some indexes can store enough information to satisfy certain queries entirely, avoiding the need to read the larger Parquet files altogether. These optimizations trade a modest increase in file or index size for significantly fewer storage operations.

Secondary Indexes

RAP also supports secondary indexes for querying across multiple dimensions (e.g., buyer ID, seller ID). These are managed at the serving layer, allowing new access paths without altering data pipelines. This flexibility is vital for evolving application requirements without costly ETL or data transformations. Hash-based indexes support exact lookups, while sorted indexes enable range queries. Techniques like Z-ordering and Hilbert curves can further enhance data locality for multi-dimensional secondary lookups.