When dealing with API calls that should only happen once (idempotent calls), you might still need retries if the first

Hey everyone, I'm working on a system where we have several internal services that need to talk to each other using REST APIs. We want to make sure operations are idempotent on the server side, like using an idempotency key header. But on the client side, what's the best way to handle retries? Should we just keep retrying forever? What's a good strategy for backing off between retries? And are there any common mistakes to watch out for when retrying operations that should be idempotent anyway?