API Gateway versus calling services directly: When should you use which?

So, I've been mulling over how we should handle communication between our frontend clients and the backend microservices. We've talked about using an API Gateway for everything coming in. It's got perks, you know, like handling authentication, limiting how many requests go through, and combining them. But honestly, sometimes it seems like just another step, and calling services directly might be easier for some things. What have you guys found? When do you absolutely need an API Gateway, and when do you go with direct calls or just a simpler Backend for Frontend pattern?