How to choose acknowledgement timing, retries, visibility timeouts and dead-letter handling for Python workers processing important backend work.
Choose delivery semantics explicitly, acknowledge only after durable progress and make every retry safe to repeat or reconcile.
Name the failure window
If a worker acknowledges before committing durable state, a crash can lose the job. If it commits and crashes before acknowledgement, the queue will usually redeliver it. That is at-least-once delivery, and it requires idempotent processing.
Exactly-once effects are rarely provided by the transport alone. Combine a durable operation key, a state transition and an outbox or inbox record where duplicates would be expensive.
Bound retries and poison messages
Retry transient failures with a bounded attempt count and backoff. Permanent validation errors should not occupy the queue indefinitely; move them to a dead-letter or review state with enough context to recover safely.
Visibility timeouts must exceed expected work but remain finite. Heartbeats or leases can extend them for long jobs, but a stuck worker should eventually become detectable and recoverable.
Operate the queue as a product surface
Monitor age of oldest message, processing latency, retry counts, dead-letter volume and worker capacity. A queue can be accepting work while user-visible latency is already unacceptable.
Test worker termination during each side effect, duplicate delivery and a dependency outage. The evidence should show what is committed, what is retried and what a human must repair.
Evidence to take into review
- Acknowledgement timing matches durable progress.
- Operations are idempotent or have a reconciliation path.
- Retries, leases and dead-letter handling are bounded.
- Queue age and retry outcomes have actionable metrics.
- Worker crashes are exercised during integration tests.
Explore all engineering notes.
Use PRODUCTION-7 to connect this concern with the other dimensions of a trustworthy backend.
View all articles Get the checklist