Designing Privacy-Preserving Scam Detection and Federated Analytics for WhatsApp
This article details WhatsApp's Scam Alert feature, an optional on-device machine learning model designed to detect potential scam messages while strictly adhering to end-to-end encryption. It highlights the architectural decisions, including on-device processing, user control, and a confidential federated analytics pipeline, to ensure user privacy and data security.
Read original on Meta EngineeringCore Design Principles for Privacy-Preserving AI
WhatsApp's Scam Alert system is built on a set of stringent design principles to ensure user privacy, particularly when integrating an ML model for content classification. The primary goal is to provide a security feature without compromising the foundational guarantee of end-to-end encryption. This necessitates that all sensitive operations remain client-side, with no automatic data transmission to servers.
- On-device only: The ML model and all message data processing occur locally on the user's device. No message content leaves the device for classification.
- No automatic reporting: WhatsApp servers cannot initiate sharing of user data. User action is explicitly required to report potential scams.
- User control: The feature is entirely optional and can be toggled by the user. Users decide how to act on scam warnings and can mark chats as trusted.
Architecture of the Scam Alert Feature
The Scam Alert leverages advances in on-device machine learning, allowing for accurate text classification without significant performance or battery impact. The model, small and effective, is trained on reported scam patterns and performs probabilistic classification based on conversational structure and linguistic signals. Warnings are displayed client-side and are not visible to the other party.
Architectural Enforcement of Privacy
The article emphasizes that privacy principles are architecturally enforced and independently verifiable. This means the system is designed to *prevent* data leakage by construction, rather than relying solely on policy.
Confidential Federated Analytics for Model Improvement
To improve the ML model over time while maintaining privacy, WhatsApp employs a confidential federated analytics pipeline. This system is crucial for collecting telemetry (e.g., warning counts, user actions) without compromising individual user data. It relies heavily on Trusted Execution Environments (TEEs) and differential privacy.
- On-Device Data Minimization: Raw signals (message content) never leave the device. Only aggregated counts (e.g., warning frequency, user action types) are sent.
- Confidential Processing: Aggregated metrics are processed within TEEs (Confidential Virtual Machines). Data is encrypted between the client and TEE, ensuring neither WhatsApp nor Meta can access individual metrics.
- Secure Aggregation: Individual device metrics are merged into running aggregates within the TEE. Only aggregated statistics, above a minimum cohort size and with differential privacy noise applied, are made available. This adds carefully calibrated noise to ensure individual data cannot be inferred.
- Verifiable Transparency: The system provides in-app capabilities for users to review shared data and privacy parameters. Additionally, the CVM image binary and source code of privacy-relevant components are publicly available for independent security research and verification.