Netflix TimeSeries Tiered Storage: Kafka/Flink and Native Cassandra Cold Reads

Managing petabytes of time-series data, particularly when most queries target recent data but historical data must remain accessible, presents significant cost and operational challenges. Netflix tackled this by introducing a tiered storage solution for their TimeSeries platform, aiming to optimize storage costs while preserving performance and API consistency.

TimeSeries Abstraction and Immutable Windows

The core of Netflix's TimeSeries platform is the abstraction that partitions datasets into "Time Slices," each covering a specific time window. A crucial optimization is the concept of immutable windows: once a time slice crosses a write boundary, it becomes immutable, enabling safe application of further optimizations like data tiering. Metadata for these time slices is maintained by a dedicated control plane, which also routes read/write traffic.

Solution 1: Streaming with Kafka and Flink to S3

The initial tiered storage solution used a familiar streaming pattern. Data was simultaneously written to Cassandra (hot storage) and streamed via Apache Kafka and Flink to S3 (cold storage). To optimize S3 writes, Flink workers hash-partition time series IDs into buckets, batching events into Parquet files grouped by event time.

• Compaction Job: A daily job merges small, hourly Parquet files into optimized daily files. These files feature small row groups (4-8 MB) for efficient range reads, event sorting by time within row groups to eliminate brute-force scans, and Bloom filters cached in EVCache for efficient query planning and pruning.
• Read Path: The TimeSeries data plane fans out queries to both hot and cold tiers, merging results. The cold tier data plane uses compaction metadata to aggressively prune S3 reads, achieving p50 latency of ~150 ms and p99 latency of ~500 ms for a full day's data.

Solution 2: Native Cassandra Cold Storage from S3 Backups

The second, more radical solution leveraged existing Cassandra backups in S3. The goal was to read directly from these S3 backups once a Time Slice was marked for cold storage, eliminating the separate streaming pipeline. This required minimal changes to the TimeSeries read path.

• ReadOnlyCompactionStrategy (ROCS): A new Cassandra compaction strategy was introduced for immutable time slices. ROCS re-compacts SSTables once, aligning them precisely with token ranges to reduce read fan-out and improve tail latency by ensuring a single partition key exists in only one SSTable, rather than across multiple overlapping files.
• Leveraging S3 Backups: Cassandra backups in S3 consist of declarative manifests and SSTable files. The manifests catalog every S3 path, allowing Cassandra nodes to download index, metadata, and integrity files locally (excluding the large Data.db file). These local components enable efficient query planning without full data scans.
• Pluggable Memtables for S3 Reads: As of Cassandra 4.1, pluggable Memtables are used to redirect reads from "cold" tables directly to S3 backup storage. When a backup Memtable is initialized, it pulls necessary SSTable components (like index files) from S3 onto the node. Reads then parallelize S3 requests across SSTables and chunks, using a shared cache for deduplication and prefetching to minimize latency.