graceful degradation, how do you keep critical features running?

You know that feeling when a minor service hiccups, and suddenly everything else starts falling apart in ways you'd never expect? I've been trying to get better at thinking ahead about graceful degradation. What do you usually do, or what patterns do you use, to make sure the really critical parts of your system keep running smoothly, even if other bits are struggling or totally offline? Got any war stories or tips from experience?