How to design Python background work for at-least-once delivery using idempotency, bounded retries, explicit states and operational evidence.
Assume delivery can repeat, make state transitions durable, bound retries and expose work that requires human recovery.
@task
def charge_order(order_id):
order = db.get(Order, order_id)
payment.charge(order.total)
order.status = "paid"
db.commit()@task(acks_late=True)
def charge_order(command_id: UUID):
command = commands.claim(command_id)
if command.completed:
return command.result
result = payment.charge(
amount=command.amount,
idempotency_key=str(command.id),
)
commands.complete(command.id, result.reference)Assume at-least-once delivery
A worker can complete external work and crash before acknowledging the message. The broker then delivers it again. Exactly-once execution is not a property to assume; safe repetition must be part of the operation design.
Persist a command identity and use it as the idempotency key for local state and supporting external providers. Protect claims and completion with atomic updates so two workers cannot both own the same transition.
Classify failure before retrying
Timeout and temporary unavailability may be retryable. Invalid input, missing permission and rejected business state usually are not. Retrying every exception wastes capacity and delays visibility of permanent failure.
Use bounded attempts, exponential backoff with jitter and a final failed state. Preserve enough safe context for an operator to understand and replay the work after correction.
Operate the queue as a product
Measure queue age, execution latency, success, retry and terminal failure by job type. Depth alone cannot distinguish healthy burst absorption from work that is permanently stuck.
Deploy workers compatibly with messages already in flight. Version message schemas or use additive evolution, and keep handlers able to process the oldest message that can still exist in retention.
Evidence to take into review
- Every task is safe under duplicate delivery.
- Retryable and permanent failures are distinguished.
- Retries have bounded attempts and jittered delay.
- Terminal failures are visible and recoverable.
- Message compatibility is part of deployment review.
Explore all engineering notes.
Use PRODUCTION-7 to connect this concern with the other dimensions of a trustworthy backend.
View all articles Get the checklist