Designing TitanMQ: A Hybrid Message Queue Learning from Kafka, RabbitMQ, and ZeroMQ

The article introduces TitanMQ, a novel message queue built from scratch to address the compromises often made when choosing between established systems like Kafka, RabbitMQ, and ZeroMQ. Each of these systems excels in certain areas but lacks in others, forcing engineers to often combine them or build custom layers. TitanMQ's goal is to unify high throughput, flexible routing, and robust persistence within a single architecture.

Core Architecture Overview

The Storage Engine: Kafka-inspired Append-Only Commit Log

Inspired by Kafka, TitanMQ's storage layer utilizes an append-only commit log for high write performance. This design leverages sequential disk writes, which are significantly faster than random writes, by never modifying data in place. Each topic-partition is managed through a series of `LogSegment` files. When a segment reaches a configurable size, a new one is rolled, ensuring efficient disk utilization and management.

public synchronized long append(TitanMessage message) throws IOException {
    ByteBuffer data = message.serialize();
    if (activeSegment.size() + data.remaining() > maxSegmentSize) {
        rollNewSegment();
    }
    long offset = nextOffset.getAndIncrement();
    activeSegment.append(offset, data);
    return offset;
}

For fast read access, TitanMQ employs a sparse in-memory index within each `LogSegment`, mapping offsets to file positions. This index allows for O(log n) lookups using binary search. While increasing memory usage per partition, this trade-off is accepted for significantly lower read latency, a critical factor for high-performance messaging systems.

Routing Engine: RabbitMQ's Flexibility with Kafka's Foundation

A major divergence from Kafka, TitanMQ integrates a flexible routing engine akin to RabbitMQ's exchange model. This allows messages to be routed to different queues or consumers based on various criteria, eliminating the need for application-level routing or multiple topics for complex scenarios. TitanMQ supports four exchange types:

• Direct Exchange: Routes messages based on an exact routing key match.
• Topic Exchange: Supports wildcard pattern matching (e.g., `*.error`, `order.#`) for more dynamic routing.
• Fanout Exchange: Broadcasts messages to all bound queues, similar to publish-subscribe.
• Content-Based Exchange: (Implicitly mentioned as an extension) This type of exchange would route messages based on inspecting the message payload itself, offering advanced filtering capabilities.