Idempotent Scan Ingestion with Redis SETNX

When two workers receive the same physical scan at the same instant — one from the scanner’s first transmission, one from its retry after a missed ACK — some single authority has to decide which of them is allowed to write the dispense to the ledger and which must stand down. In a single process a lock would do; across a fleet of stateless ingestion workers it cannot, because there is no shared memory to lock. The authority has to live outside the workers, in a store every worker can reach, and the claim has to be atomic: no window in which two workers both read “not seen” and both proceed. Redis gives exactly that primitive in a single round-trip — SET key value NX PX <ttl> sets the key only if it does not already exist and attaches an expiry in one atomic operation. The first worker to run it wins the key and commits; every other worker sees the key already present and suppresses its redundant delta. This recipe implements the exact, distributed store that the parent reference, Real-Time Barcode-Scan Deduplication, falls through to whenever a scan needs an authoritative duplicate decision.

This is the authoritative half of the deduplication design. The probabilistic pre-filter in Deduplicating Barcode Scans with Bloom Filters makes the common case cheap, but it can never declare a duplicate — only this exact claim can. Correctness for controlled substances rests here.

Problem framing

The deduplication gate needs a decision that is exact — no false positives, no false negatives — and distributed, so that any worker in the fleet that happens to receive a retransmission reaches the same verdict. An in-process set cannot do this across workers, and a read-then-write against a database opens a race: two workers both read “absent” before either writes. The atomic conditional write is the whole point. SET key value NX PX <ttl> collapses the check and the set into one operation the store executes indivisibly, so exactly one caller can ever observe the transition from absent to present. That caller is the first writer; it commits. Everyone else is, by definition, a duplicate.

The PX <ttl> half matters as much as the NX half. The key must not live forever — a genuine re-dispense of the same product minutes later must be allowed through — so the claim carries an expiry sized to the physical duplicate window. Inside the window, retransmissions collapse onto the held key; after it, the key is gone and a legitimately new movement of the same product proceeds.

Prerequisites & environment

• Python 3.11+ and redis (redis-py) 4.x or newer, which exposes nx= and px= keyword arguments on Redis.set.
• A Redis deployment you trust for the window. For a single-window idempotency guard a standard primary with persistence is adequate; the failover caveat in the gotchas below governs how durable it must be.
• A deterministic scan hash. The key must be reproducible from the scan alone so that a retransmission computes the same key. This is the composite device · payload · window digest defined by the parent reference; the same bytes in, the same key out, on every worker.
• Regulatory context. The exactly-once accounting standard of 21 CFR § 1304.21 is the reason the claim must be atomic — two workers committing the same dispense is a double-count and a recordkeeping failure. The retention standard of 21 CFR § 1304.04 is why the suppression of a duplicate is itself logged, not silently discarded.

Implementation: an atomic first-writer-wins claim

The store below wraps a single SET … NX PX into a claim that returns True only for the worker that first created the key within its TTL. Every other concurrent or later arrival inside the window gets False and is suppressed. The TTL is derived from the deduplication window so the claim expires precisely when a new movement of the same product should be allowed through.

from __future__ import annotations

import hashlib
import json
import logging
from dataclasses import dataclass
from datetime import datetime, timezone

import redis

logger = logging.getLogger("pharmacy.scan.redis")

_KEY_PREFIX = "scan:idem:"


@dataclass(frozen=True)
class ScanClaim:
    """A deterministic idempotency identity for one physical scan. The digest
    is computed from device, payload, and the quantized window so that a
    retransmission reproduces the identical key on any worker."""
    facility_uuid: str
    scanner_id: str
    ndc: str
    lot_number: str
    quantity: int
    scan_type: str
    window_index: int

    def key(self) -> str:
        material = "|".join((
            self.facility_uuid, self.scanner_id, self.ndc, self.lot_number,
            str(self.quantity), self.scan_type, str(self.window_index),
        ))
        digest = hashlib.sha256(material.encode("utf-8")).hexdigest()
        return f"{_KEY_PREFIX}{digest}"


class RedisIdempotencyStore:
    """Exact, distributed first-writer-wins guard over Redis SET NX PX.
    Satisfies the exactly-once accounting standard of 21 CFR § 1304.21."""

    def __init__(self, client: redis.Redis, window_seconds: int = 3, guard_ms: int = 500) -> None:
        self._redis = client
        # TTL = window + a small guard so a duplicate straddling a window
        # boundary is still caught, but not so long a real re-dispense is blocked.
        self._ttl_ms = window_seconds * 1000 + guard_ms

    def claim(self, claim: ScanClaim) -> bool:
        key = claim.key()
        # NX: create only if absent. PX: expire after the window.
        # A truthy reply means THIS caller created the key => first writer.
        created = self._redis.set(key, "1", nx=True, px=self._ttl_ms)
        outcome = "committed" if created else "suppressed"
        self._audit(key, outcome)
        return bool(created)

    def _audit(self, key: str, outcome: str) -> None:
        # PHI-free: the key is a SHA-256 digest; no operator or patient data.
        logger.info(json.dumps({
            "event": "scan_claim",
            "key": key,
            "outcome": outcome,
            "ttl_ms": self._ttl_ms,
            "at": datetime.now(timezone.utc).isoformat(),
        }, sort_keys=True))


if __name__ == "__main__":
    r = redis.Redis(host="localhost", port=6379, db=0)
    store = RedisIdempotencyStore(r, window_seconds=3)
    c = ScanClaim(
        facility_uuid="F" * 36, scanner_id="SCN-04", ndc="00093721410",
        lot_number="A1207", quantity=1, scan_type="dispense", window_index=587_402_113,
    )
    print(store.claim(c))   # True  -> first writer commits the dispense
    print(store.claim(c))   # False -> retransmission suppressed as duplicate

The method returns a plain boolean the gate acts on directly: True means “you are the first writer, commit the delta,” and False means “this is a duplicate, suppress it and acknowledge.” Because SET … NX is atomic, two workers running claim at the same microsecond cannot both receive True — Redis serializes them, and exactly one observes the absent-to-present transition. That is the whole idempotency guarantee, delivered in one round-trip.

Verification & testing

The tests assert the two properties that matter: only the first of several identical claims wins, and the claim expires after its TTL so a later genuine movement is not blocked. A fake Redis keeps the test self-contained; against a real server the same assertions hold.

import time
import pytest


class FakeRedis:
    """Minimal SET NX PX stand-in with millisecond expiry for tests."""
    def __init__(self) -> None:
        self._store: dict[str, float] = {}   # key -> expiry epoch (seconds)

    def set(self, key, value, nx=False, px=None):
        now = time.monotonic()
        exp = self._store.get(key)
        if exp is not None and exp <= now:
            del self._store[key]              # lazily expire
        if nx and key in self._store:
            return None                       # already held => not first writer
        self._store[key] = now + (px / 1000.0 if px else 0)
        return True


def _claim():
    return ScanClaim("F" * 36, "SCN-04", "00093721410", "A1207", 1, "dispense", 587_402_113)


def test_only_first_writer_wins():
    store = RedisIdempotencyStore(FakeRedis(), window_seconds=3)
    c = _claim()
    assert store.claim(c) is True      # first send commits
    assert store.claim(c) is False     # retry suppressed
    assert store.claim(c) is False     # third transmission suppressed


def test_claim_expires_after_window():
    store = RedisIdempotencyStore(FakeRedis(), window_seconds=0, guard_ms=50)
    c = _claim()
    assert store.claim(c) is True
    time.sleep(0.1)                    # let the 50 ms TTL lapse
    assert store.claim(c) is True      # a later genuine movement is allowed

The audit line emitted on each claim records the outcome and the SHA-256 key, never the raw scan or any identity:

INFO pharmacy.scan.redis {"at": "2026-07-16T14:03:11.402Z", "event": "scan_claim", \
     "key": "scan:idem:9f2c...e41a", "outcome": "suppressed", "ttl_ms": 3500}

Joining every audit line that shares a key reconstructs the physical pick: exactly one committed, and one suppressed per retransmission — the evidence that the redundancy was observed and handled under 21 CFR § 1304.04.

Gotchas & compliance pitfalls

• TTL too short. If the expiry is shorter than the real retransmission delay of the hardware, a retry arriving after the key has already expired reads “absent,” wins its own claim, and double-commits the dispense — a phantom overage. Size the TTL to the window plus a guard that covers the worst-case retry latency, never to a bare guess.
• TTL too long. If the expiry outlives the interval between two genuine dispenses of the same product from the same scanner, the second real movement collides with the still-held key and is wrongly suppressed — an under-count. The window must sit below the fastest realistic re-dispense interval. This is the tension the window width resolves, and it is why the TTL is a tuned parameter.
• Failover can lose the key. Redis replication is asynchronous, so a primary can acknowledge a SET and then crash before the write reaches a replica; the promoted replica has no key, and a retransmission wins a second claim. For controlled-substance accountability, back the guard with a durable configuration — WAIT-acknowledged replication, AOF persistence with appendfsync always, or a consensus-backed store — or reconcile against the ledger’s own uniqueness constraint as a backstop so a lost key cannot silently double-count.
• Key collision from a weak identity. If the key omits a distinguishing dimension — say it drops scanner_id — two genuinely distinct dispenses of the same product from two scanners in the same window collide on one key, and the second is suppressed as a phantom duplicate. Include every dimension that makes two events actually different: device, full payload, and window.
• Clock skew across workers and devices. The window_index must be derived from a normalized UTC time, computed the same way on every worker, or two views of the same event quantize into different windows and produce different keys — defeating the claim. Normalize timestamps at capture and compute the window from the ingestion-normalized time, not raw device clocks.

Frequently Asked Questions

Related

• Up: Real-Time Barcode-Scan Deduplication — the parent reference where this exact store backs the dedup gate.
• Deduplicating Barcode Scans with Bloom Filters — the probabilistic pre-filter whose “possibly seen” verdict falls through to this claim.
• Barcode Scan Log Routing Logic — the routing subsystem whose idempotency step this pattern implements.
• Fallback Routing for Offline Sync — how idempotency keys survive an outage so replays converge to one commit.