When should a growing web app start using a CDN?

Hey everyone, I'm hoping you can share some of your experiences. Our web app has grown quite a bit, and we're starting to worry about performance. Right now, we're serving all our static files straight from our own servers. I'm curious, when do you usually decide it's time to add a Content Delivery Network, or CDN? Are there particular traffic numbers, how slow things are getting, or certain kinds of files that make you pull the trigger? And what have been the downsides you've noticed, especially with getting it set up at first and the costs afterwards?