So, when's the best time to add a CDN to your setup?

I've been thinking a lot about making our content delivery better for everyone, especially with users all over the world. Right now, everything comes straight from our main app servers. It's fine for now, but I can see lag becoming a real problem as we get bigger. So, when does a CDN really start to pay off, even for things that aren't just static files like images? Are there certain ways people use our site, or where they're located, or even specific kinds of dynamic content that make you say, "Yep, we need a CDN now"? I'd love to hear what you've run into and any tricky parts you've discovered.