When does caching turn into more trouble than it's worth?

Everyone talks about how great caching is for speed and taking pressure off the database. But I'm wondering about the other side of that coin. When does caching actually cause more headaches than it fixes? I'm picturing things like stale information, making cache invalidation super complicated, or just the hassle of running a whole caching system. What are your real-world tales or specific examples of when caching went sideways? And how did you figure out it was time to either cut back or completely change your caching approach?