Cache stampede prevention: probabilistic vs mutex-based approaches

we've hit cache stampedes a few times now, where a popular cache key expires and a huge surge of requests all try to regenerate it from the backend simultaneously, often taking down downstream services. we've explored the mutex-based approach, which serializes access to regeneration, but it can introduce latency under heavy load. probabilistic early expiration seems to avoid this by allowing some requests to regenerate before expiry, but it doesn't completely eliminate the thundering herd if many clients try this. another option is background cache warming for the most popular keys. i'm curious about real-world trade-offs. what strategies have teams found most effective in preventing cache stampedes without adding unacceptable latency or over-complicating the caching layer for general use?