Lean System Design: Building a URL Shortener with Realistic Scale
This article challenges common system design interview assumptions by analyzing the real-world scale of major URL shortener services like Bitly and TinyURL. It demonstrates that a single-machine solution using PostgreSQL or SQLite and an auto-incremental ID to Base64URL encoding can meet and even exceed the actual needs of most real-world applications, emphasizing a "lean" approach over over-engineered distributed solutions often discussed in interviews.
Read original on Dev.to #systemdesignChallenging Interview Assumptions
System design interviews often present hypothetical scale requirements (e.g., 100 million URLs generated per day) that lead candidates to design highly distributed, complex systems. This article critically examines these requirements against the actual operational scale of leading URL shortening services like Bitly and TinyURL. It reveals that even the largest players operate at significantly lower throughputs (e.g., ~93 writes/sec, ~3820 reads/sec for Bitly), suggesting that many interview problems might prompt over-engineering for real-world scenarios.
Typical Interview Solutions vs. Lean Design
Traditional interview solutions for URL shorteners often involve complex strategies like hashing long URLs with collision resolution or using distributed unique ID generators (e.g., Snowflake IDs) to achieve high scalability and ensure short, unique IDs. While these approaches are technically sound for massive scale, the article argues they might be unnecessary for most practical applications. The "lean system design" philosophy advocates for simpler solutions that are appropriate for the actual expected load.
Common Interview Architectures
- Hashing with Collision Resolution: Generates a hash from the long URL, stores it in a relational database, and handles collisions by retrying or appending data. Requires efficient indexing (e.g., B-tree or Bloom filter) for `EXISTS` checks.
- Distributed Unique ID Generator: Uses a service like Snowflake to generate globally unique IDs, which are then encoded (e.g., Base62) to form the short URL. This approach inherently minimizes collisions and simplifies ID management in distributed environments.
A Practical, Single-Machine Approach
The article proposes a "real-world" design for a URL shortener using a single machine with a standard relational database (PostgreSQL or SQLite). Instead of complex hashing or distributed ID generation, it leverages a simple primary auto-incremental ID, which is then encoded using Base64URL. This approach is significantly simpler and easier to operate.
Performance Benchmarks
A basic Go implementation running on low-tier hardware (Intel Celeron N5105 with HDDs) demonstrated impressive performance, achieving ~2540 writes/sec and ~15869 reads/sec on a pre-filled database of 3 billion records. These numbers far exceed the real-world observed loads of major URL shortening services, validating the viability of a single-machine approach for most scenarios.
Availability in a Lean Setup
While a single machine setup offers perfect consistency, availability is a concern if the machine goes down. However, the article points out that even enterprise-grade services like Bitly promise 99.9% availability, which still allows for significant downtime (over 40 minutes per month). For many applications, this level of downtime might be acceptable, especially when considering the operational simplicity and cost savings of a single-machine architecture. Deployments can be fast, and data migrations can often be handled with compatible changes without service interruption.