when does caching complexity outweigh its benefits, honestly

so, been mulling over caching a lot, you know, with all the recent chats about CDNs and database caching. it's pretty obvious caching can make things way faster and take some load off. but seriously, when does wrestling with cache invalidation, fighting stale data, and setting up distributed caching get to be more trouble than it's worth? are there certain points or situations where you'd say 'nah, don't bother with a caching layer', even if it looks like a good idea for speed at first glance