When does a CDN become something you absolutely need, not just something that's cool to have?

I've been thinking about when it makes sense to set up a Content Delivery Network. We're sending out assets all over the world now, and it works, sure, but some people definitely experience slow load times. So, when does the cost and hassle of a CDN become too much compared to the good stuff like faster pages and less work for our main servers? Are there certain traffic numbers, where users are located, or what kind of stuff we're sharing that just screams 'yes, get a CDN'?