Kafka consumer lag: monitoring and alerting strategies

Kafka consumer lag is one of those metrics that can quickly spiral out of control if not properly monitored and alerted on. We had an incident where a consumer group fell behind by millions of messages over a weekend, completely unnoticed until downstream data pipelines started complaining about missing or stale data. Our previous static threshold (e.g., 'alert if lag > 10,000 messages') was problematic because it would constantly false alarm during legitimate batch processing or system restarts. We need a more intelligent strategy. I've heard about monitoring the *rate of change* of lag, rather than just the absolute lag itself, or setting dynamic thresholds based on message production rates. What are your best practices for monitoring Kafka consumer lag effectively? What alerting strategies have you found reduce noise while still catching critical issues, especially in environments with highly variable message production rates and consumer processing speeds?