When does it make sense to spend the time building your own memory allocator?

Meta going back to jemalloc made me think about if custom memory allocators are really worth it. Sure, big systems get clear performance boosts, especially when you tailor them for certain tasks and hardware. But man, it means digging deep into C/C++ code, which can cause tons of problems and is a pain to keep up. For most stuff, the usual library allocators are just fine. So, when does a special memory allocator actually become a good idea, and not just some costly optimization that doesnt give you much back? Is it mainly about how fast things respond, how much data it handles, memory getting fragmented, or a mix of all that? Id love to hear what others have experienced or their rules for deciding this kind of thing.