Designing a High-Throughput, Low-Latency Notification System: Lessons from Real-World Implementation
This article details the architectural decisions and trade-offs involved in building a notification system capable of handling 10 million messages daily with strict latency and data loss requirements. It emphasizes that practical, hands-on building with real constraints is crucial for mastering system design, offering insights beyond theoretical knowledge.
Read original on Dev.to #systemdesignThe article highlights the critical difference between theoretical system design knowledge and the practical application of making architectural decisions under real-world constraints. It argues that true system design skill comes from repeatedly building systems, encountering failures, and iterating on solutions, rather than just memorizing textbook patterns or abstract concepts.
Core Requirements for a High-Scale Notification System
- Multi-channel Delivery: Support for various notification types (push, SMS, email).
- High Throughput: Process 10 million alerts daily.
- Low Latency: Sub-200ms latency for message delivery.
- Zero Data Loss: Absolute guarantee that no notification is lost.
- Asynchronous Processing: All delivery must be non-blocking.
- Cost Efficiency: Operate within a fixed hourly budget.
Architectural Components and Decisions
The author settled on an architecture designed to meet the stringent requirements, focusing on decoupling and resilience. Key components and decisions included:
- API Gateway: Routes incoming notification requests to appropriate services.
- Dedicated Notification Services: Separate services or workers for each channel (push, SMS, email) to isolate failures and manage channel-specific logic.
- Message Queue (Kafka/MSK): Used to decouple the ingestion of notifications from their actual delivery. This ensures asynchronous processing and acts as a buffer against spikes in traffic or slow downstream services. MSK (Managed Streaming for Kafka) was chosen for its scalability and reliability.
- Separate Delivery Workers: Each notification channel (e.g., SMS, Email) has its own pool of workers to prevent slow channels from bottlenecking faster ones. This is crucial for maintaining overall system latency and preventing cascading failures.
- Dead Letter Queue (DLQ): Captures messages that fail to be delivered after exhausting all retry attempts. This is vital for the "zero data loss" requirement, allowing for inspection, manual intervention, and re-processing.
- Monitoring and Logging: Comprehensive monitoring and logging infrastructure is essential to prove data loss guarantees and quickly identify delivery issues or bottlenecks.
The Importance of Cost Constraints
The article emphasizes that a fixed budget significantly alters design decisions. It forces engineers to think beyond simply adding more resources and to critically evaluate the necessity and efficiency of each component, leading to more optimized and realistic architectures.