Dead letter queues: patterns for handling failed messages gracefully

we've got a dead letter queue that currently holds about 50,000 unprocessed messages, and frankly, nobody really looks at them. it's a catch-all for anything that fails, which isn't a great strategy. we need a more structured approach to dlqs. i'm thinking about a proper retry mechanism with exponential backoff for transient errors, but also a way to classify permanent failures so they don't clog the queue indefinitely. a monitoring dashboard showing error types and volumes would be essential, along with a tool for bulk replay of corrected messages. what patterns have people found effective for managing dlqs at scale, beyond just 'send it to a queue and forget about it'?