When does going serverless just make things harder than they need to be?

Hey everyone, I've been doing some projects lately, and serverless functions felt like the right move for a bit. But then I ran into problems. You know, those cold starts, getting stuck with one vendor, and having too many tiny functions to keep track of. It really got me thinking, when does serverless just make things harder instead of easier? Are there particular projects or situations where you'd totally avoid it? Even for stuff that seems like it should be stateless? I'm curious about your actual experiences.