24 discussions in the community
we've got a dead letter queue that currently holds about 50,000 unprocessed messages, and frankly, nobody really looks at them. it's a catch-all for a...
we're running into complexities with event ordering in kafka. for most of our use cases, partitioning by `userId` gives us good enough ordering guaran...
our cto is pushing to use kafka for literally everything, including sending emails and generating reports, citing standardization benefits. while i ag...
Ensuring reliable event publishing from within database transactions is a common challenge for maintaining data consistency across services. We need t...
the concept of exactly-once delivery in distributed systems feels like a holy grail. textbooks always say it's impossible, yet kafka advertises exactl...
event ordering in kafka is always a fun one. we partition by `userId` to ensure order for user-specific events, which works well for things like 'item...
we're constantly dealing with event schema evolution across our data pipelines. adding new fields is usually fine if consumers are resilient, but remo...
we're designing a new event-driven service that needs to process about 10k messages per second, requiring at-least-once delivery, and the ability to r...
Kafka consumer lag is one of those metrics that can quickly spiral out of control if not properly monitored and alerted on. We had an incident where a...
Event schema evolution is a recurring nightmare for us. We started with simple JSON messages, and it was fine until we began adding, removing, or rena...
the holy grail of distributed systems always seems to be 'exactly-once delivery,' but textbooks typically say it's impossible. kafka, however, often a...
We're building out a new event-driven system, and the choice of message queue is a big debate right now: Kafka, RabbitMQ, or SQS. We expect around 10,...
our cto is pushing to use kafka for pretty much everything, including what i feel are simple async job processing tasks like sending email notificatio...
we're designing an event-driven system that needs to handle about 10k messages per second, requiring at-least-once delivery, replayability, and about ...
we had a situation last month where our kafka consumer lag for a critical service grew to millions of messages, completely unnoticed for several hours...
Our CTO is really keen on standardizing everything on Kafka, which is great for our high-throughput data pipelines. However, this includes pushing asy...
we're dealing with a common distributed systems problem: needing to atomically update a database record and publish an event related to that change. t...
the outbox pattern keeps coming up in discussions about reliably publishing events from within a database transaction. the core problem is needing to ...
we've accumulated about 50k messages in our dead letter queues across various services, and honestly, nobody's really looking at them. it's become a d...
Our dead letter queues (DLQs) have become something of an unmanaged graveyard. We have services that send failed messages to DLQs, which is great in t...
