How much caching is too much?

Hey everyone, I've been dealing with some systems recently, and caching is coming up a lot. We've got in-memory caches, distributed caches, CDNs... it's quite a bit. Caching is great for performance, sure, but I'm starting to wonder if we can sometimes go overboard. What have you all found when it comes to cache invalidation problems, adding complexity, or even just the expense of managing all those different caching layers? How do you know when enough is enough?