Data Durability Trade-offs in Storage Engines
This article explores the critical concept of data durability in storage systems, delving into what it truly means when a database acknowledges a write. It breaks down the four layers a write traverses and the significant "fsync tax" associated with ensuring data persistence against power loss, contrasting it with process crashes. The piece then details the Write-Ahead Log (WAL) as a fundamental pattern for achieving durability and efficient recovery, highlighting its ubiquitous use in various database systems.
Read original on Dev.to #systemdesignThe Illusion of "Saved": Understanding Data Durability
When a database replies with a `200 OK` after a write operation, it's often assumed the data is safely on disk. However, this article highlights that the reality is more nuanced. Data can reside in various volatile memory layers before reaching non-volatile storage, exposing it to loss during specific failure events. The core challenge is to guarantee durability without sacrificing write and read performance, a tension every storage engine must resolve.
Acknowledged vs. Durable
A common mistake, as illustrated by a real-world analytics SaaS incident, is acknowledging writes before they are truly durable. This can lead to silent data loss that only surfaces much later, as un-flushed in-memory buffers are lost during unexpected power-class events.
The Four Layers of a Write and the fsync Tax
A `write()` syscall does not immediately place data on disk. Instead, it moves through four distinct layers, with only the last one being truly non-volatile:
- App buffer (Process RAM): Lost on both process crash and power loss.
- OS page cache (Kernel RAM, dirty): Survives process crashes but lost on power loss.
- Disk controller cache (RAM on the SSD): Lost on power loss.
- NAND / platter (Non-volatile media): The only truly durable layer.
The distinction between `write()` and `fsync()` is crucial. While `write()` is fast (microseconds), `fsync()` forces data down to NAND and blocks until confirmed, introducing a significant performance overhead known as the fsync tax (0.1-2 ms on SSDs, 5-15 ms on HDDs). Skipping `fsync` can make writes appear faster but creates a "durability window" during which data is vulnerable to power loss, though it might survive a simple process crash.
The Write-Ahead Log (WAL) for Robust Durability
The Write-Ahead Log (WAL) is a foundational pattern for achieving robust data durability and efficient recovery. The core principle is simple: write the change to an append-only log *first*, ensure that log entry is durably on disk (via `fsync`), then acknowledge the write. The actual data structure update happens asynchronously in the background.
SET balance=100
① append to WAL (sequential, at the end of the file)
② fsync(wal_fd) (one cheap flush → durable)
③ return 200 OK (now it is safe to say "saved")
...later, in the background...
④ apply to the real data structure- Append-only for efficiency: Writing to the end of a single file is a sequential operation, which is significantly faster than random writes to update scattered pages in a data structure like a B-tree.
- Simplified Recovery: In case of a crash, recovery involves simply replaying the committed entries from the WAL in order. Incomplete or torn writes in the WAL are easily detected and discarded.
- Immutability and Order: The WAL preserves the history of changes. Data is never modified in place within the log; space is reclaimed through checkpointing or compaction. This guarantees that log order is the source of truth.
WAL in Real-World Systems
The WAL pattern is central to many widely used systems, including PostgreSQL (pg_wal), SQLite (WAL journal mode), RocksDB/LevelDB (write-ahead log for memtable), Kafka (the entire log *is* the database), and Redis (Append-Only File).
Group commit is a key optimization for WAL-based systems. Instead of performing an `fsync` for every individual write, multiple writes arriving within a short time window (e.g., 1-10 ms) are batched, appended to the log, and then a *single* `fsync` is performed for the entire batch. This amortizes the high cost of the `fsync` syscall across many writes, significantly increasing overall write throughput, albeit with a slight increase in latency for individual requests.