Building a Hybrid AI Moderation Platform for Real-Time Marketplaces
This article outlines DoorDash's journey in building SafeChat, an AI-powered content moderation platform designed to operate at scale within a real-time marketplace environment. It details an architectural pattern that combines fast, cheaper internal models for obvious cases with more expensive, nuanced LLM scoring for complex decisions, enabling graduated actions and significantly reducing safety incidents. The system evolved into a content-agnostic platform adaptable to various moderation use cases beyond its initial scope.
Read original on InfoQ ArchitectureThe Challenge of Real-Time AI Moderation at Scale
DoorDash faces the challenge of moderating millions of daily messages, calls, and images exchanged between users, Dashers, and merchants in real time. The key requirement is to classify content as safe or unsafe within a fraction of a second to avoid disrupting the user experience, while simultaneously minimizing operational costs. Initially, a pure Large Language Model (LLM) approach was considered but quickly discarded due to high latency (2-10 seconds per call) and prohibitive costs for 4 million daily calls.
Hybrid AI Architecture: Cheap First, Smart Later
DoorDash adopted a hybrid architectural pattern to overcome the LLM limitations. This involved a multi-layered approach:
- Data Understanding: Months were spent analyzing existing data to quantify the percentage of unsafe messages (found to be a small single-digit percent). This insight was crucial for designing an efficient system.
- Layer 1: Fast, Cheap Internal Classifier: A small, in-house ML model (responding in <100ms, no per-call cost) acts as a preliminary filter. Its primary job is to quickly identify "obviously safe" content. Messages classified as safe are immediately delivered.
- Layer 2: LLM Multi-Axis Scoring: Messages not confidently classified as safe by Layer 1 (less than 10% of total messages) are forwarded to a more expensive LLM. Crucially, the LLM is not asked for a simple boolean (safe/unsafe) but rather to score the content across multiple severity axes (e.g., threatening, profane, sexual). This provides a "knob" instead of a "flag", allowing for more granular decision-making.
- Graduated Actions: Based on the multi-axis scores from the LLM, the system takes graduated actions. This ranges from censoring low-severity content, blocking mid-severity messages, to canceling orders and warning offenders for high-severity threats.
Design Principle: Score, Don't Boolean
When using AI/ML models for moderation or decision-making, especially with LLMs, prefer asking for scores or classifications across multiple dimensions rather than a simple true/false. Scores enable more flexible, graduated responses, future-proofing for new categories, and adjustable thresholds, making the system more robust and adaptable.
Platform Evolution and Content Agnosticism
After successfully reducing verbal abuse incidents by 50%, the SafeChat system evolved into a content-agnostic moderation platform. The underlying "cheap first, expensive second, then graduated action" pattern proved valuable for various use cases beyond chat, including moderating profile pictures, names at signup, food reviews, and even fraud detection. This platform approach avoids rebuilding similar moderation logic for each new content type (images, voice), using appropriate "cheap layers" like commercial vision APIs for images or voice-to-text with immediate action for voice calls, while leveraging the same scoring engine.