api gateway vs. direct service calls, when to switch?

ive been using an api gateway for our microservices for a while and its been great for centralizing auth and rate limiting but lately im wondering if were hitting its limits for internal service to service communication direct calls between services seem simpler and potentially faster in some cases whats your experience when did you decide it was time to bypass the gateway for certain internal interactions or conversely when did you stick with it even for those what were the trade offs you observed