Apache Kafka Deep Dive

What Makes Kafka Different

Apache Kafka is not a traditional message queue. It is a distributed commit log — an append-only, ordered, durable sequence of records. Unlike RabbitMQ or SQS where messages are deleted after consumption, Kafka retains messages for a configurable retention period (days, weeks, or forever with tiered storage). Consumers track their own position using offsets. This fundamentally changes the replay and reprocessing story.

Core Architecture

Topics and Partitions

A Kafka topic is divided into one or more partitions. Each partition is an ordered, immutable log. Producers write to partitions (round-robin by default, or keyed by a partition key). Consumers read from partitions sequentially using an offset — an integer index into the log.

Partitions are the unit of parallelism. With 12 partitions and 12 consumer instances in a group, you get 12x parallel consumption. You cannot have more active consumers than partitions in a group — extra consumers sit idle.

Consumer Groups and Offsets

A consumer group is a set of consumer instances that share the work of reading a topic. Kafka assigns each partition to exactly one consumer in the group. If an instance fails, Kafka rebalances — redistributing that partition to another instance. Each group independently tracks its own committed offset per partition in an internal Kafka topic called `__consumer_offsets`.

Replication and Durability

Each partition has a replication factor (typically 3). One broker is the leader for that partition — it handles all reads and writes. The other brokers are followers that replicate from the leader. If the leader fails, a follower is elected. The `acks` producer setting controls durability:

Exactly-Once Semantics

Kafka supports idempotent producers (enable with `enable.idempotence=true`) to prevent duplicate writes from retries. For end-to-end exactly-once, Kafka transactions let you atomically write to multiple partitions and commit offsets in a single atomic operation. This enables read-process-write pipelines (Kafka Streams style) with exactly-once guarantees.

// Kafka producer with exactly-once semantics (Java)
Properties props = new Properties();
props.put("bootstrap.servers", "kafka:9092");
props.put("enable.idempotence", "true");
props.put("acks", "all");
props.put("transactional.id", "my-transactional-id");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();

try {
    producer.beginTransaction();
    producer.send(new ProducerRecord<>("output-topic", key, value));
    // Atomically commit processed offset + new record
    producer.sendOffsetsToTransaction(offsets, consumerGroupId);
    producer.commitTransaction();
} catch (Exception e) {
    producer.abortTransaction();
}

Kafka's Retention Model vs Traditional Queue

This is where Kafka fundamentally differs. Traditional queues delete messages after acknowledgment. Kafka retains messages for a configured period (default 7 days). This enables:

• Replay — Re-process all events from the beginning when you deploy a new consumer service
• Multiple independent consumers — Each consumer group has its own offset; 10 different teams can read the same topic independently
• Time-travel debugging — Rewind a consumer to a specific timestamp to diagnose production issues
• Event sourcing — The Kafka topic IS the source of truth; the database is a projection

When to Choose Kafka