At scale, retries are not an edge case. They are part of the normal operating model. A worker can crash, a downstream service can time out, and a message can be delivered again.
Why idempotency matters
Idempotency gives the system permission to retry aggressively without turning temporary failures into duplicate side effects.
A reliable shape
A production-ready worker path usually needs:
- A stable idempotency key
- Clear job state transitions
- Bounded retries
- Dead-letter handling
- Metrics for queue depth, age, throughput, and failure reason
Once those controls exist, autoscaling becomes safer because more workers increase capacity without multiplying ambiguity.