when do you really need caching, and when is it just a bonus?

Lately, I've been on a few projects where adding caching felt like we'd just tack it on later. But honestly, looking back, some of those systems would've been way faster and more stable right from the jump if we'd just thought about caching sooner. What about you? When do you decide to build caching into the infrastructure and plan from the beginning? Is it just about how many reads you expect, or does stuff like latency needs or how complicated it is to get the data play a role too?