When does caching actually make things slower or more complex than they're worth?

Everyone talks about caching like it's some magic fix for speed, but I wonder when it actually causes more headaches than it's worth. I'm thinking about those times when trying to keep the cache fresh turns into a total mess, or when managing a big distributed cache just isn't worth the effort for how people are using your site. Has anyone actually seen performance get worse because they added a cache, or had it make things way more complicated for no real upside? What kinds of setups or user behaviors are especially prone to running into problems with caching?