how much is too much for api rate limiting

We're seeing some crazy traffic spikes on our public APIs lately. We've put in some basic rate limiting, but I'm curious about the breaking point. When does tough rate limiting start annoying actual users or making the API a pain to work with, instead of just stopping the bad actors? How do you guys figure out the right balance here?