Mitigating Cross-Tenant Data Exposure in Multi-tenant Container Platforms
This article details a vulnerability found in Cloudflare's multi-tenant container environment where residual data from previous tenants could be exposed. It explains the underlying storage mechanism (Linux device mapper thin provisioning) and how a configuration oversight (skip_block_zeroing) allowed data fragments to persist, leading to a cross-tenant data leak. The mitigation steps involved reconfiguring storage to zero out blocks and a comprehensive cleanup of existing container disks and caches.
Read original on Cloudflare BlogCloudflare's incident report highlights a critical security vulnerability in their multi-tenant Containers and Sandboxes platform. This case study is highly relevant to system design as it delves into the intricacies of secure multi-tenancy, storage isolation, and the potential pitfalls of low-level infrastructure configurations. Understanding such vulnerabilities is crucial for architects designing secure cloud platforms or shared environments.
The Vulnerability: Residual Data Exposure
The vulnerability stemmed from the use of Linux device mapper thin provisioning (dm-thin) for container root disks. In a multi-tenant setup, physical storage blocks are reused across different customer workloads. Cloudflare's configuration included `skip_block_zeroing`, which meant that when a previously used 64 KiB block was reassigned, a smaller write (e.g., 4 KiB) would only overwrite a portion, leaving the remaining 60 KiB potentially accessible with data from the previous tenant. This allowed a Workers Paid account to recover residual disk blocks from other tenants on the same host.
Key Takeaway for System Designers
In multi-tenant systems, strict isolation at all layers is paramount. This includes ensuring that physical storage is securely wiped or zeroed before being reallocated to a new tenant to prevent data remnants from previous users. Overlooking low-level storage configurations can lead to severe security breaches.
Storage Allocation and Exploit Mechanism
- dm-thin Provisioning: Each container receives a writable root disk via `dm-thin`, with physical storage allocated on demand.
- Shared Block Pool: When a thin volume is deleted, its physical blocks are returned to a shared pool serving multiple customer accounts.
- `skip_block_zeroing`: This crucial configuration option prevented newly allocated blocks from being zeroed, leading to residual data.
- Exploitation: Researchers wrote a small 4 KiB block into free space, which triggered the allocation of a 64 KiB physical block. Due to `skip_block_zeroing`, the remaining 60 KiB could contain data from a previous tenant, which could then be read.
Mitigation Strategy
- Configuration Change: Immediately removed `skip_block_zeroing` from the `dm-thin` pool configuration, restoring default behavior to zero new allocations.
- Fleet-wide Cleanup: Retired all running container disks and removed cached image snapshots created before the mitigation. This involved draining hosts, restarting VMs, and clearing host image caches to ensure all disks and layers were recreated with zeroed allocations.