Scalable Privacy-Preserving Anonymization with Core Mondrian for Data Analysis
Airbnb's Project Lighthouse employs privacy-preserving anonymization techniques to analyze user experience disparities without compromising individual privacy. This article introduces `project-lighthouse-anonymize`, an open-source Python library implementing a scalable k-anonymity algorithm based on an extended Mondrian approach. It also details a framework for measuring and validating the quality of anonymized data to ensure statistical validity for downstream analyses.
Read original on Airbnb EngineeringProject Lighthouse is an initiative by Airbnb to measure and mitigate potential disparities in user experiences, particularly concerning perceived race data. A cornerstone of this project is its commitment to privacy, ensuring that sensitive attribute data is never linked to individual accounts. This privacy-by-design approach necessitated the development of robust anonymization techniques, which are now open-sourced through the `project-lighthouse-anonymize` Python library.
Foundational Privacy Methodology: p-sensitive k-anonymity
The initial methodology for Project Lighthouse, established in 2020, adopted k-anonymity as the technical privacy model. This model aims to prevent sensitive attribute disclosure by ensuring that for any given record, there are at least k-1 other records indistinguishable from it with respect to certain quasi-identifier attributes. The system also leverages p-sensitive k-anonymity, which further protects sensitive attributes within these k-anonymous groups by ensuring that no single sensitive value is overly dominant.
Core Mondrian: Scalable k-Anonymity Algorithm
The `project-lighthouse-anonymize` library's core is the Core Mondrian algorithm, an extension of the classic Mondrian algorithm for k-anonymity. This algorithm is designed for scalable partition-based anonymization of large datasets. Key architectural enhancements include:
- Extensible architecture: Utilizes the Strategy Pattern to support k-anonymity and allow for easy integration of future privacy models.
- Parallel processing: Employs a hybrid recursive-queue execution model, combining immediate recursive processing for smaller data partitions with queue-based parallel processing for larger ones, optimizing performance.
- NaN-pattern pre-partitioning: Efficiently accommodates missing values within the dataset during the anonymization process.
- Dynamic suppression budget management: Provides flexibility in managing how much data is suppressed to meet k-anonymity requirements.
Architectural Design Choice: Strategy Pattern
The use of the Strategy Pattern in Core Mondrian highlights a common system design principle: extensibility. By abstracting the privacy model logic, the system can easily support new anonymization algorithms or variations without requiring significant changes to the core processing engine. This modularity is crucial for evolving systems in sensitive domains like privacy.
Measuring Data Quality for Anonymized Data
A critical challenge with anonymization is ensuring the utility and statistical validity of the resulting data. The article introduces a comprehensive framework for measuring data quality, including three primary metrics and an empirical validation methodology. This allows analysts to confidently use anonymized data for statistical conclusions, bridging the gap between privacy protection and data utility.