Designing a URL Shortening Service

Core Functional & Non-Functional Requirements

A URL shortening service must handle input of long URLs and output unique short URLs, facilitate redirection from short to original URLs, and optionally support custom aliases and expiration times. Key non-functional requirements include low redirect latency (e.g., < 50ms), high availability, and an architecture optimized for a read-heavy workload.

High-Level Architecture Overview

The proposed architecture utilizes a URL Service as the main entry point, backed by Redis for caching and PostgreSQL for persistent storage. Analytics are handled asynchronously using Kafka for event streaming and a dedicated Analytics Consumer that ingests data into ClickHouse, an OLAP database suitable for time-series and aggregate analytics.

+----------------+ Client ----------> | URL Service | +----------------+ | | Redis Cache PostgreSQL | Redirect | Async Event (Kafka) | +----------------+ | Analytics | | Consumer | +----------------+ | ClickHouse (Analytics DB)

URL Creation and Redirection Workflows

1. URL Creation: A client sends a POST request with the long URL. The system inserts the long URL into PostgreSQL, generates a unique ID, encodes this ID to a short code (e.g., using Base62), updates the database with the short code, and returns the shortened URL.
2. URL Redirection: Upon receiving a GET request for a short URL, the system first checks Redis cache. If found, it immediately returns a 301 redirect. If a cache miss occurs, it queries PostgreSQL, updates the cache, and then performs the 301 redirect. Analytics events for redirection are pushed to Kafka asynchronously.

Distributed Short Code Generation

For large distributed systems, generating unique and short IDs is critical. The article suggests using a Snowflake-like ID generation algorithm combined with Base62 encoding. Snowflake IDs offer a distributed, time-ordered, and unique ID generation scheme, preventing collisions and enabling high throughput. Base62 encoding converts these numeric IDs into URL-friendly alphanumeric strings, optimizing for length and character set compatibility.

| 1 bit | 41 bits | 10 bits | 12 bits |
|-------|---------|----------|----------|
| Sign | Timestamp | Worker ID | Sequence |