Message queues for async job processing: is Kafka overkill?
Maria Kumar
·345 views
Our CTO is really keen on standardizing everything on Kafka, which is great for our high-throughput data pipelines. However, this includes pushing async job processing like sending email notifications, generating reports, and image processing into Kafka topics. While Kafka *can* do this, it feels like overkill and potentially not the right tool for the job.
For these types of tasks, lighter-weight message queues or job queues like BullMQ (for Node.js), Celery (for Python), or managed services like AWS SQS seem more appropriate. They often provide better features for job management (retries, delays, prioritization), error handling, and simpler consumption patterns for these specific use cases. Using Kafka for everything adds operational complexity, and the features it excels at (like replayability and high throughput event streaming) aren't necessarily critical for sending a single email. Is there a point where trying to standardize on one technology becomes a detriment, or am I underestimating Kafka's suitability for these kinds of generic async job queues?
10 comments