async processing for api calls, when do we jump in?

Hey everyone, I've been mulling over when it makes sense to start using async processing for API endpoints. I mean, for tasks that take ages, it's totally obvious you should, but what about when the normal response time is pretty good, like under half a second, but sometimes it just crawls? Would you go ahead and add a message queue just in case, or would you wait until it's actually a big issue? I'm really interested in hearing what you all have done and what data you check before making the switch to async.