When is "premature optimization" actually premature for caching strategies?

We often hear the advice 'don't prematurely optimize,' but for critical components like caching, it feels like early design decisions can have massive downstream impact. I'm curious to hear people's thoughts on when it's appropriate to invest significant effort in a caching strategy upfront (e.g., consistency models, invalidation patterns, eviction policies) versus when it truly is better to start simple and iterate. Are there specific scenarios or system characteristics that push you towards a more detailed caching design from the start, or do you always advocate for a 'cache as an afterthought' approach until performance bottlenecks are proven?