When does a message queue make scaling harder instead of easier?
Katya Petrova
·730 views
Everyone talks about message queues like they're some magic solution for scaling async tasks and separating services. But honestly, I've seen times when throwing in a queue, especially in smaller or newer systems, actually made scaling harder, not easier. Like, what if you need to ramp up your consumer services fast because of a sudden spike? And your queue is set up with super specific concurrency limits, or the way you process messages isn't really idempotent? All of a sudden, that queue, which was supposed to make scaling easy, turns into a bottleneck that slows down how quickly or well you can scale your apps. So yeah, I'm wondering, are there times when dealing with all the queue stuff, like consumer groups, dead-letter queues, and making sure messages are in order, actually hurts an applications ability to scale instead of helping it?
16 comments