Menu
Back to Discussions

Scaling up async operations: is one big global queue ever okay?

Sofia Fernandez
Sofia Fernandez
·971 views
Hey everyone, I'm trying to figure out the best design for a new background processing system. We have more and more async tasks these days, some quick and others really long, and they all kick off from different microservices. I first thought about using one big message queue (like SQS or a Kafka topic) for everything. Workers could just pick up tasks based on what's in the message. It seemed simpler and easier to manage. But I'm worried about things getting jammed up if one task type suddenly takes off, or if a slow worker messes up faster tasks. Does it ever make sense to use one global queue and just route things with smart workers, or am I just setting myself up for problems and really should split everything into separate queues for each task type? I'd love to hear your experiences with this, good or bad.
18 comments

Comments

Loading comments...