So, when should you use a CDN versus caching on the server itself?

We're dealing with a lot of static assets (images, JS, CSS) for our web app, and the load is growing. We've optimized our server-side caching for dynamic content, but static assets are a different beast. I'm wondering about the tipping point for introducing a Content Delivery Network (CDN). At what point does the complexity and cost of a CDN outweigh just sticking with well-tuned server-side caching (like Redis or Varnish) for these assets? What are your experiences with this trade-off?