when does caching become a performance bottleneck itself

We all know how much caching helps speed things up when reading data. We use Redis a ton for session info, API answers, and even some calculated stuff. But I've been wondering, when does the caching layer itself turn into a problem? Are there typical ways things go wrong, or situations where dealing with cache invalidation, the time it takes to reach the cache, or even when lots of things try to access the cache at once, actually make things slower than they should be? I'd love to hear what you guys have tried and what works for you to get around this.