So, when should you really start thinking about sharding your database?

we're starting to see some serious read latency creep into our primary user table, and writes arent far behind. We arent at a scale where its unmanageable, but its definitely getting noticeable. Ive been reading up on sharding, but it feels like a big architectural shift. What are the telltale signs youve hit the point where sharding is less of an 'if' and more of a 'when'? When did you all make the jump, and what were the biggest challenges?