Strategies for handling backpressure in message queue systems?

Hey everyone, I'm working on a system where a downstream service can sometimes get overwhelmed by the volume of messages coming from an upstream producer. We're using a message queue (Kafka, in this case), and I'm looking for effective strategies to handle this backpressure without losing messages or crashing the consumer. Beyond just scaling out consumers, what are some common patterns or techniques you've successfully implemented? I'm thinking about things like dynamic consumer throttling, producer-side rate limiting, or even more advanced queue management strategies. Any war stories or best practices would be super helpful!