caching when does the complexity outweigh the benefits

We all know how much faster things get when you use caching, right? Whether it's in your computer's memory, spread across different servers, or just on the user's browser, it really speeds up how quickly stuff responds and takes the pressure off your backend. But, caching isn't all sunshine and rainbows. You've got to deal with invalidation, making sure things are consistent, using up more memory, and the headache of managing all those cache servers. So, when does all the work and the chance of new bugs become too much, and it's just not worth the speed boost anymore? Have you ever decided not to cache something, or even taken out caching that you put in before?