When is serverless architecture the wrong choice?

We've been exploring serverless options for new services, and while the benefits of reduced operational overhead and scaling are clear, I'm curious about the scenarios where it might actually be a detriment. For example, cold start times for latency-sensitive applications or the potential for vendor lock-in are often cited. Are there specific types of workloads or architectural patterns where serverless has proven to be more problematic than beneficial in your experience? What hidden costs or complexities have you encountered that made you reconsider a serverless approach for a particular project?