InfoQ Architecture·August 19, 2026

On-Device ML for Scam Detection with Privacy-Preserving Analytics in WhatsApp

WhatsApp is implementing an on-device machine learning feature, "Scam Alert," to detect scam messages from non-contacts. The system design prioritizes user privacy by performing message classification locally, while leveraging confidential computing, differential privacy, and transparency ledgers for performance measurement and secure model distribution.

AI & ML Infrastructure Security Distributed Systems

Read original on InfoQ Architecture

WhatsApp's new "Scam Alert" feature exemplifies a robust system design for integrating on-device machine learning with stringent privacy requirements. The core challenge is to identify scam patterns effectively without compromising user message content or privacy during model evaluation and performance analytics.

Architectural Overview: Separation of Concerns

The architecture distinctly separates message classification from performance measurement and model distribution. Message classification occurs entirely on the user's device, using a downloaded ML model to analyze conversational structure and linguistic signals from non-contact messages. This ensures that sensitive message content never leaves the device for scam detection.

Privacy-Preserving Analytics Pipeline

Local Aggregation: On-device, warning events and user actions (block, report, continue, trust) are aggregated into counts.
Oblivious HTTP Relay: These aggregated metrics are transmitted via Oblivious HTTP (OHTTP) using anonymous credentials, obfuscating the source of the data.
Confidential Computing: Data is processed within confidential virtual machines (trusted execution environments). Clients verify the code running in this environment and privacy parameters before data transmission, ensuring computations are isolated and protected.
Differential Privacy & Cohort Thresholds: Before making statistics available to WhatsApp, the aggregation process applies minimum cohort thresholds and differential privacy to ensure only approximate, population-level statistics are derived, preventing the re-identification of individuals.

💡

Why OHTTP and Confidential Computing?

OHTTP and confidential computing are critical for preserving user privacy in analytics. OHTTP prevents servers from linking telemetry data to individual users, while confidential computing provides a hardware-protected environment for processing sensitive aggregated data, ensuring that even the cloud provider cannot access the plaintext data or computations.

Secure Model Distribution and Verification

Model distribution is treated as a security boundary. Each model version, along with its SHA-256 hash, is published to a third-party append-only transparency ledger. Clients verify this ledger entry, the model's signature, freshness, and hash before loading a model, preventing malicious model injections. Model downloads also use anonymous credentials and OHTTP, with experiment assignment occurring locally to prevent server-side targeting of specific model variants to individual users.

This architectural approach builds upon Meta's PAPAYA federated analytics system, adapting it for consumer messaging where telemetry data itself is considered sensitive. It highlights a strong commitment to privacy by design, particularly for features leveraging AI/ML in sensitive contexts.

on-device machine learningprivacy-preserving analyticsconfidential computingdifferential privacyfederated learningmobile securityOHTTPscam detection

Comments

Loading comments...

Architecture Design

Design this yourself

Design a secure, privacy-preserving messaging platform that incorporates on-device machine learning for real-time content moderation (e.g., scam or spam detection). Detail the architecture for model distribution, on-device inference, and the collection of anonymous, aggregated performance metrics using techniques like Oblivious HTTP, confidential computing, and differential privacy.

Practice Interview

Focus: on-device machine learning for content moderation with privacy-preserving analytics

Other design angles

· Design an end-to-end encrypted chat system and explain how on-device ML for content moderation could be integrated without compromising E2EE.· Design a federated learning system for mobile applications that collects usage patterns while ensuring strong user privacy guarantees through techniques like secure aggregation and differential privacy.· Architect an API gateway and backend system for an application that relies on client-side AI inference, focusing on how to securely manage and distribute ML models to millions of devices.