monitoring and alerting for distributed systems what's your go-to

hey everyone, i've been thinking a lot about how we monitor our distributed systems lately. with so many moving parts, services, and dependencies, it feels like keeping an eye on everything and knowing *when* to sound the alarm is getting trickier. what are your preferred tools or strategies for setting up effective monitoring and alerting? are you guys using Prometheus, Datadog, or something else? how do you decide what metrics are truly important to alert on versus just noise?