Async Retry with Exponential Back-off

Committing an inventory update across a network is a fallible operation: the database connection pool exhausts, the wholesaler API returns a 503, a TLS session drops mid-write. None of those are reasons to lose a Schedule II movement, and none are reasons to hammer a struggling downstream into collapse. The correct response is a bounded asyncio retry loop with exponential back-off and full jitter that gives way to a dead-letter path once attempts are exhausted. This recipe builds that loop, and it operates within the Async Batch Processing for Inventory Updates workflow, where the retried unit of work is an idempotent, key-guarded ledger commit.

The single rule that makes retries safe in a regulated inventory system is that every attempt must be idempotent. A retry that is not tied to a stable idempotency key does not recover a lost write — it manufactures a duplicate one, and a duplicated controlled-substance delta is a reportable discrepancy under 21 CFR § 1304.11. So this page assumes each unit of work already carries the deterministic key produced in Batch Idempotency Key Generation; the retry loop simply re-drives the same keyed commit until it lands exactly once or is quarantined with full context.

Problem framing & prerequisites

The retry loop sits between a worker picking up a batch and the atomic ledger transaction that commits it. Its job is narrow and its failure modes are specific: retry the transient faults, never the permanent ones, cap the total attempts so recovery is bounded rather than infinite, and spread the retries in time so a fleet of workers recovering from the same outage does not stampede the downstream the instant it comes back. The broader failure taxonomy — which exceptions are transient versus terminal — is owned by the Error Handling & Retry Mechanisms subsystem; this page implements the timing and hand-off mechanics for the transient class.

Environment:

• Python 3.11+ — the implementation uses asyncio, @dataclass(frozen=True), and structured logging.
• Standard library only for the loop: asyncio, random, hashlib, logging, dataclasses. Add pytest and pytest-asyncio for the verification block.
• Regulatory grounding: 21 CFR § 1304.11 (no lost or duplicated Schedule II movement), 21 CFR § 1304.04 (records reproducible for inspection), 45 CFR § 164.312(b) (audit controls). Controlled-substance work carries a higher attempt ceiling than ordinary syncs because dropping the event creates a DEA discrepancy, but the ceiling is still finite — after it, the work quarantines, it is never silently discarded.

Implementation: full-jitter back-off over an idempotent commit

The delay schedule is base * 2 ** attempt, capped at a maximum, then multiplied by a random factor in [0, 1] — full jitter. Equal-jitter and “exponential with a fixed sleep” both leave a synchronized spike where many workers wake at the same instant; full jitter flattens it, which is what protects a just-recovered database from a thundering herd. The retried callable is the keyed, idempotent commit, so re-driving it is always safe.

import asyncio
import logging
import random
from dataclasses import dataclass
from typing import Awaitable, Callable

logger = logging.getLogger("pharmacy.batch.retry")


class TransientCommitError(Exception):
    """Recoverable: connection reset, 429, 503, pool timeout — safe to retry."""


class PermanentCommitError(Exception):
    """Terminal: schema/compliance violation — retrying will fail identically."""


@dataclass(frozen=True)
class RetryPolicy:
    max_attempts: int = 7        # controlled-substance ceiling; 3–5 for ordinary syncs
    base_delay: float = 0.5      # seconds
    max_delay: float = 30.0      # cap so backoff never grows unbounded
    dlq_on_exhaust: bool = True

    def backoff(self, attempt: int) -> float:
        """Full-jitter delay for a zero-based attempt index."""
        ceiling = min(self.max_delay, self.base_delay * (2 ** attempt))
        return random.uniform(0.0, ceiling)  # full jitter: uniform in [0, ceiling]


async def commit_with_retry(
    commit: Callable[[], Awaitable[bool]],
    idempotency_key: str,
    policy: RetryPolicy,
    dead_letter: Callable[[str, str, int], Awaitable[None]],
) -> bool:
    """Drive an idempotent commit under exponential back-off with full jitter.

    `commit` MUST be idempotent (guarded by `idempotency_key`) so that a retry
    after a partially-applied write cannot double-post a controlled-substance
    delta — satisfying the no-duplicate requirement of 21 CFR § 1304.11.
    """
    last_error = "unknown"
    for attempt in range(policy.max_attempts):
        try:
            result = await commit()
            logger.info("COMMIT_OK key=%s attempt=%d", idempotency_key, attempt + 1)
            return result
        except PermanentCommitError as exc:
            # Never retry a deterministic failure — it wastes the attempt budget
            # and delays the compliance review the record actually needs.
            logger.error("COMMIT_PERMANENT key=%s reason=%s", idempotency_key, type(exc).__name__)
            await dead_letter(idempotency_key, f"permanent:{type(exc).__name__}", attempt + 1)
            return False
        except TransientCommitError as exc:
            last_error = type(exc).__name__
            if attempt + 1 >= policy.max_attempts:
                break
            delay = policy.backoff(attempt)
            logger.warning(
                "COMMIT_RETRY key=%s attempt=%d next_delay=%.3fs reason=%s",
                idempotency_key, attempt + 1, delay, last_error,
            )
            await asyncio.sleep(delay)

    # Attempts exhausted on a transient fault: quarantine, never drop.
    logger.error("COMMIT_EXHAUSTED key=%s attempts=%d reason=%s",
                 idempotency_key, policy.max_attempts, last_error)
    if policy.dlq_on_exhaust:
        await dead_letter(idempotency_key, f"exhausted:{last_error}", policy.max_attempts)
    return False

Two design choices carry the compliance weight. First, PermanentCommitError skips the loop entirely: a schema or DEA-classification failure will fail identically on every attempt, so retrying it only burns the budget and delays the pharmacist review the record actually needs. Second, exhaustion routes to dead_letter rather than returning a bare failure — a transient fault that outlives the attempt ceiling is preserved with its key and attempt count, so the movement is recoverable. The structure and replay semantics of that quarantine are the subject of Dead-Letter Queue Design for Inventory Events.

Because the retried callable is keyed and idempotent, the loop is also safe under the offline path: a commit that was partially applied before an outage, then re-driven on reconnect, is deduplicated by its key rather than double-posted.

Verification & testing

The tests must prove four behaviours: a transient fault eventually succeeds within the cap, a permanent error is not retried, exhaustion hands off to the dead-letter path, and the delay schedule grows monotonically in its ceiling. Patch asyncio.sleep so the suite does not actually wait.

import pytest

@pytest.fixture
def policy():
    return RetryPolicy(max_attempts=5, base_delay=0.5, max_delay=30.0)

@pytest.mark.asyncio
async def test_transient_then_success(policy, monkeypatch):
    monkeypatch.setattr(asyncio, "sleep", lambda _: asyncio.sleep(0))
    calls = {"n": 0}
    async def commit():
        calls["n"] += 1
        if calls["n"] < 3:
            raise TransientCommitError("503")
        return True
    async def dlq(*_): raise AssertionError("should not reach DLQ")
    assert await commit_with_retry(commit, "idem:batch:x", policy, dlq) is True
    assert calls["n"] == 3

@pytest.mark.asyncio
async def test_permanent_error_is_not_retried(policy):
    calls = {"n": 0}
    async def commit():
        calls["n"] += 1
        raise PermanentCommitError("DEA_SCHEDULE_INVALID")
    seen = {}
    async def dlq(key, reason, attempts): seen.update(key=key, reason=reason)
    assert await commit_with_retry(commit, "idem:batch:x", policy, dlq) is False
    assert calls["n"] == 1                       # exactly one attempt, no retry
    assert seen["reason"].startswith("permanent")

@pytest.mark.asyncio
async def test_exhaustion_hands_off_to_dlq(policy, monkeypatch):
    monkeypatch.setattr(asyncio, "sleep", lambda _: asyncio.sleep(0))
    async def commit(): raise TransientCommitError("reset")
    seen = {}
    async def dlq(key, reason, attempts): seen.update(reason=reason, attempts=attempts)
    assert await commit_with_retry(commit, "idem:batch:x", policy, dlq) is False
    assert seen["attempts"] == 5
    assert seen["reason"].startswith("exhausted")

def test_backoff_ceiling_grows_and_caps(policy):
    ceilings = [min(policy.max_delay, policy.base_delay * 2 ** a) for a in range(8)]
    assert ceilings[:4] == [0.5, 1.0, 2.0, 4.0]
    assert ceilings[-1] == 30.0                  # capped, never unbounded
    for _ in range(1000):
        assert 0.0 <= policy.backoff(3) <= 4.0   # full jitter stays within ceiling

A representative log line from an exhausted controlled-substance batch carries the key, attempt count, and reason — no patient data, no raw dispensing detail:

WARNING pharmacy.batch.retry COMMIT_RETRY key=idem:batch:5f1e6c2a:9f2c7d attempt=3 \
        next_delay=2.114s reason=TransientCommitError
ERROR   pharmacy.batch.retry COMMIT_EXHAUSTED key=idem:batch:5f1e6c2a:9f2c7d attempts=7 \
        reason=TransientCommitError

Gotchas & compliance pitfalls

• Retrying a non-idempotent write. Without a stable idempotency key, a retry after a partially-applied commit posts the delta twice. Tie every attempt to the key from the idempotency-key recipe; the loop must re-drive the same keyed operation, never a freshly built one.
• Thundering herd without jitter. Plain exponential back-off wakes every worker recovering from a shared outage at the same instant, re-crushing the downstream. Full jitter — random.uniform(0, ceiling) — spreads the retries; equal-jitter still leaves a synchronized floor.
• Retrying permanent errors. A schema violation or invalid DEA schedule will fail identically forever. Retrying it wastes the attempt budget and delays the pharmacist review the record requires. Classify first, retry only the transient class.
• Unbounded back-off. base * 2 ** attempt without a cap grows to minutes, then hours; a batch stuck behind it silently misses its reconciliation window. Cap the ceiling (max_delay) and cap the attempts (max_attempts) — bounded recovery, not infinite.
• Silent exhaustion. Returning a bare failure on exhaustion drops the movement. Always hand off to the dead-letter path so a Schedule II event is preserved for redrive, never discarded.
• Sleeping the event loop wrong. Use await asyncio.sleep(delay), never time.sleep, inside a coroutine — a blocking sleep stalls every other task on the loop, including the heartbeats that keep the broker lease alive.

Frequently Asked Questions

Related

• Async Batch Processing for Inventory Updates — parent workflow whose keyed commits this loop re-drives.
• Batch Idempotency Key Generation — the deterministic key every retry must be tied to.
• Error Handling & Retry Mechanisms — the transient-versus-terminal taxonomy this loop consumes.
• Dead-Letter Queue Design for Inventory Events — the quarantine an exhausted batch is handed to.