replication lag how do you manage it in production

So, I've been running into replication lag in my distributed database setup lately. It's kinda annoying, you know? It feels small until it causes a stale read and then BAM, you've got problems. How do you guys keep an eye on and fix replication lag? Any specific ways you build your systems or tools you really like that help keep it from getting out of hand? I'd love to hear about your bad experiences and what worked for you.