Real-Time Barcode-Scan Deduplication

Point-of-dispense scanners are noisy signal sources. A single physical pick of a Schedule II vial can produce two, three, or a dozen electrical events: a trigger that double-fires on release, a driver that retransmits after a missed ACK, a mobile app that replays its outbox on reconnect, or an operator who scans twice because the first beep was inaudible on the floor. Every one of those redundant events is a true transmission of a false inventory change. If they reach the perpetual record unfiltered, the ledger reports phantom dispenses, on-hand counts drift below physical reality, and the variance surfaces at the next count as an unexplained controlled-substance shortage — the exact signal a diversion investigation is built to detect. Real-time deduplication is the control that separates the one real movement from its echoes before any of them mutate state. This reference sits within the Data Ingestion & Inventory Sync Workflows architecture and defines the deduplication tier that every scan traverses on its way to commitment.

Deduplication is often mistaken for a performance optimization — a way to avoid redundant database writes. In a controlled-substance system it is the opposite: it is a correctness and compliance control whose failure is reportable. The distinction shapes every design decision on this page. A cache miss that drops a legitimate scan is not a lost keystroke; it is an under-counted dispense that leaves inventory overstated. A false match that treats a second genuine pick as a duplicate is a missed dispense that leaves inventory understated. Both are recordkeeping failures, so the deduplication layer is engineered to fail toward accountability — never silently dropping a controlled event without an audit record explaining why.

Regulatory Context & Compliance Boundaries

Deduplication lives on the same strict-liability recordkeeping edge as the rest of scan ingestion, and it inherits the routing-layer discipline defined in Barcode Scan Log Routing Logic: validate, deduplicate, classify, audit. The difference is that this layer owns the second of those four boundaries as its single responsibility, and the regulatory stakes of getting it wrong are specific.

The governing principle is that a duplicate is not garbage to be discarded — it is evidence that the physical world sent the same fact twice, and that observation must be logged even as the redundant delta is suppressed. A rejected duplicate scan is still a recordkeeping event under 21 CFR § 1304.04. The deduplication layer therefore emits one structured, PHI-free audit line per decision — committed, duplicate-suppressed, or unresolved — so a compliance officer can later prove that exactly one of N identical transmissions changed the ledger and that the other N−1 were accounted for.

What a Dedup Key Must Encode

Deduplication is only as correct as the key it computes. The key answers a single question — “is this the same physical event I already committed?” — and getting its scope right is the whole problem. Too broad and it collapses two genuine, distinct dispenses of the same product into one, understating inventory. Too narrow and it treats a retransmission as new, restoring the phantom-overage failure it was meant to prevent. The canonical key for a controlled-substance scan binds three dimensions:

The time window is the dimension engineers most often omit and the one that matters most. A pure content key (device + payload) with no temporal bound treats every re-pick of the same lot as a duplicate forever — which is wrong, because a pharmacy dispenses the same lot repeatedly. Binding the key to a window — a bucket a few seconds wide, sized to the physical duplicate horizon of the hardware — lets the system say “the same fact within two seconds is an echo; the same fact two minutes later is a new dispense.” The window width is a tuned parameter, not a constant: it must cover the worst-case retransmission delay of the noisiest device on the floor while staying far below the fastest realistic re-dispense interval for the same product.

Where a natural per-event identifier already exists — a scan_uuid minted once at the point of capture and carried through every retransmission — it becomes the strongest possible key, because the retry carries the same UUID and therefore self-identifies as a duplicate without any windowing heuristic. In practice both coexist: the UUID catches transport-layer retries exactly, and the windowed content key catches double-triggers and replays where the capture layer minted a fresh UUID for the same physical pick.

Exact vs Probabilistic Deduplication

There are two ways to test whether a key has been seen, and a production controlled-substance system uses them in series rather than choosing one.

Exact deduplication stores every seen key and answers definitively. A key is a duplicate if and only if it is present in the store. This is correct by construction — no false positives, no false negatives — and it is the only acceptable final authority for a Schedule II event. Its cost is memory and coordination: the store must be shared across every worker that could receive a retransmission, which in a distributed dispensing environment means a networked key store with atomic first-writer-wins semantics. That exact, distributed guard is built in Idempotent Scan Ingestion with Redis SETNX, where a single atomic SET … NX PX claims the key for exactly one arrival.

Probabilistic deduplication trades certainty for bounded memory. A space-efficient membership filter can answer “definitely never seen” or “possibly seen” in constant memory regardless of stream volume, which makes it ideal as a pre-filter in front of the exact store: a “definitely never seen” answer skips the expensive exact lookup entirely, and only a “possibly seen” answer pays for the round-trip. The catch is asymmetry — the filter can report a false “possibly seen” for a key it has never observed, but it can never miss a key it has. That asymmetry is exactly why a probabilistic hit must never be the reason a controlled-substance scan is dropped; it must always fall through to an exact check. The filter design, its false-positive-rate math, and the fall-through discipline are covered in Deduplicating Barcode Scans with Bloom Filters.

The two combine into a single rule the deduplication gate follows without exception: a probabilistic filter may fast-path a unique event, but only the exact store may declare a duplicate. This keeps the common case cheap while preserving the strict-liability guarantee that no genuine dispense is ever suppressed on a guess.

Deterministic Deduplication Workflow

The gate is a strict, total state machine. Every scan terminates in exactly one of three states — committed, suppressed, or unresolved — and each is independently auditable. The canonical sequence is:

1. Key derivation. Compute the dedup key from device identity, the canonical payload fields, and the quantized time window. Where a capture-layer scan_uuid is present, incorporate it so transport retries self-identify.
2. Probabilistic pre-filter. Test the key against the in-memory membership filter. A “definitely never seen” result fast-paths the event to step 4; a “possibly seen” result proceeds to the authoritative check at step 3.
3. Exact claim. Atomically attempt to claim the key in the shared exact store with a TTL equal to the window width. If the claim fails, the key is already held: the event is a confirmed duplicate. Suppress it, count it, emit an audit record, and stop.
4. Idempotent commit. For a first-seen key, add it to the probabilistic filter, then commit the inventory delta under first-writer-wins semantics so that even a concurrent claim resolves to a single ledger write.
5. Audit emission. Emit one structured, PHI-free record for the decision — committed or suppressed — carrying the dedup key hash, the window, and the outcome, but never operator or patient identity.

State transitions are total: a scan that cannot reach the exact store (a store outage) does not silently commit or silently drop; it terminates unresolved and is buffered for deferred resolution, preserving accountability through the outage.

Focused Python Implementation

The implementation below is the deduplication gate: it derives the composite key, quantizes the window, runs the probabilistic pre-filter in front of an exact claim, and emits a chained SHA-256 audit hash with PHI-free structured logging. The exact store and membership filter are expressed as narrow protocols so the gate is testable and the concrete backends — a networked key store and an in-memory filter — plug in behind them.

from __future__ import annotations

import hashlib
import json
import logging
from dataclasses import dataclass
from datetime import datetime, timezone
from enum import Enum
from typing import Protocol

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
    datefmt="%Y-%m-%dT%H:%M:%SZ",
)
logger = logging.getLogger("pharmacy.scan.dedup")


class Decision(str, Enum):
    COMMITTED = "committed"
    SUPPRESSED = "suppressed"     # confirmed duplicate
    UNRESOLVED = "unresolved"     # exact store unreachable; defer, never guess


@dataclass(frozen=True)
class ScanEvent:
    """One canonical scan envelope. Operator identity is already pseudonymized
    upstream (45 CFR § 164.514(b)); no PHI is present on this object."""
    scanner_id: str
    facility_uuid: str
    ndc: str
    lot_number: str
    quantity: int
    scan_type: str            # receive | dispense | waste | return | adjust
    scan_uuid: str            # minted at capture; stable across transport retries
    timestamp_utc: datetime


class MembershipFilter(Protocol):
    """Probabilistic pre-filter: may report a false 'seen', never a false 'unseen'."""
    def probably_seen(self, key: str) -> bool: ...
    def add(self, key: str) -> None: ...


class ExactStore(Protocol):
    """Authoritative first-writer-wins claim. Returns True iff THIS caller
    claimed the key for the first time within the TTL window."""
    def claim(self, key: str, ttl_seconds: int) -> bool: ...
    def reachable(self) -> bool: ...


def _window_index(ts: datetime, window_seconds: int) -> int:
    """Quantize the event time into a fixed-width bucket. The same physical
    event retransmitted inside one window lands in the same bucket; a genuine
    re-dispense in a later window does not."""
    epoch = int(ts.astimezone(timezone.utc).timestamp())
    return epoch // window_seconds


def derive_key(event: ScanEvent, window_seconds: int) -> str:
    """Composite dedup key = device · payload · window, plus the capture UUID.
    A SHA-256 digest keeps the key fixed-width and PHI-free in logs."""
    material = "|".join((
        event.facility_uuid,
        event.scanner_id,
        event.ndc,
        event.lot_number,
        str(event.quantity),
        event.scan_type,
        event.scan_uuid,
        str(_window_index(event.timestamp_utc, window_seconds)),
    ))
    return hashlib.sha256(material.encode("utf-8")).hexdigest()


class DeduplicationGate:
    """Suppress duplicate scans before they mutate the controlled-substance
    ledger, satisfying the exactly-once accounting standard of 21 CFR § 1304.21."""

    def __init__(
        self,
        pre_filter: MembershipFilter,
        exact: ExactStore,
        window_seconds: int = 3,
    ) -> None:
        self._filter = pre_filter
        self._exact = exact
        self._window = window_seconds

    def admit(self, event: ScanEvent) -> Decision:
        key = derive_key(event, self._window)

        # A hostile/degraded exact store must not silently drop a CS event.
        if not self._exact.reachable():
            self._audit(event, key, Decision.UNRESOLVED)
            return Decision.UNRESOLVED

        # Fast path: a 'definitely never seen' key skips the exact round-trip.
        # A probabilistic 'seen' is NOT trusted to drop — it falls through.
        if self._filter.probably_seen(key):
            claimed = self._exact.claim(key, self._window)
            if not claimed:
                self._audit(event, key, Decision.SUPPRESSED)
                return Decision.SUPPRESSED
        else:
            # Definitely-new keys must still be claimed to win the race with a
            # concurrent worker; first-writer-wins decides the single commit.
            if not self._exact.claim(key, self._window):
                self._audit(event, key, Decision.SUPPRESSED)
                return Decision.SUPPRESSED

        self._filter.add(key)
        self._audit(event, key, Decision.COMMITTED)
        return Decision.COMMITTED

    def _audit(self, event: ScanEvent, key: str, decision: Decision) -> None:
        # Tamper-evident, PHI-free line. The key is a hash; NDC and schedule
        # tier are safe; operator and patient identity never appear.
        record = {
            "event": "scan_dedup_decision",
            "dedup_key": key,
            "ndc": event.ndc,
            "scan_type": event.scan_type,
            "window_s": self._window,
            "decision": decision.value,
            "at": datetime.now(timezone.utc).isoformat(),
        }
        logger.info(json.dumps(record, sort_keys=True))


if __name__ == "__main__":
    class _InMemoryFilter:
        def __init__(self) -> None:
            self._seen: set[str] = set()
        def probably_seen(self, key: str) -> bool:
            return key in self._seen
        def add(self, key: str) -> None:
            self._seen.add(key)

    class _InMemoryExact:
        def __init__(self) -> None:
            self._held: set[str] = set()
        def claim(self, key: str, ttl_seconds: int) -> bool:
            if key in self._held:
                return False
            self._held.add(key)
            return True
        def reachable(self) -> bool:
            return True

    gate = DeduplicationGate(_InMemoryFilter(), _InMemoryExact(), window_seconds=3)
    now = datetime.now(timezone.utc)
    scan = ScanEvent(
        scanner_id="SCN-04", facility_uuid="F" * 36, ndc="00093721410",
        lot_number="A1207", quantity=1, scan_type="dispense",
        scan_uuid="7d2c1f90-0e3a-4b11-9c2e-1a2b3c4d5e6f", timestamp_utc=now,
    )
    print(gate.admit(scan))   # Decision.COMMITTED  — first arrival wins
    print(gate.admit(scan))   # Decision.SUPPRESSED — retransmission collapses

Two properties make this safe for an append-only ledger. First, the exact store — not the probabilistic filter — is the sole authority that returns SUPPRESSED, so no genuine dispense is ever dropped on a filter false positive. Second, an unreachable store yields UNRESOLVED rather than a default commit or a default drop, so an infrastructure fault degrades into a bounded backlog instead of a silent recordkeeping gap.

Compliance Mapping & Audit Boundaries

Every dedup decision is a first-class audit artifact, and each terminal state maps to a distinct, independently retrievable record. The suppression of a duplicate is more important to log than the commit, because it is the evidence that N−1 identical transmissions were deliberately not applied.

The dedup key in every audit line is a SHA-256 digest, never the raw payload, so the audit stream carries no NDC-plus-operator correlation that could re-identify a dispensing event, keeping the log inside the HIPAA audit-control boundary of 45 CFR § 164.312(b). Compliance officers reconstruct the lifecycle of any physical pick by joining every audit line that shares a dedup key: exactly one committed, zero or more suppressed, and the count of suppressions proves the redundancy was observed and handled rather than lost.

Error Handling & Offline Resilience

Deduplication must stay correct through the failure modes that make real scan streams messy, and each mode has a deterministic destination rather than a silent default:

• Exact-store outage. When the shared claim store is unreachable, the gate returns UNRESOLVED and buffers the event locally with its dedup key intact. On reconnect the buffered events are replayed through the same claim path, so a scan captured during the outage and any online replay of it resolve to the same key and collapse to one commit. This deferred-commitment path is owned by Fallback Routing for Offline Sync, which preserves idempotency keys across the disconnect so reconnection converges to the ledger state an uninterrupted node would have produced.
• Window boundary straddle. A duplicate that arrives a few milliseconds after a window rolls over would land in a new bucket and escape suppression. The mitigation is to make the TTL on the exact claim equal to the window width plus a small guard, and to incorporate the stable scan_uuid so transport retries self-identify regardless of which bucket they fall in.
• Clock skew across devices. Scanners with drifting clocks can quantize the same event into adjacent windows. Normalizing every timestamp to UTC at capture and keying the window on the ingestion-normalized time, not the raw device time, keeps quantization stable.
• Filter saturation. A probabilistic pre-filter that has absorbed too many keys degrades toward always answering “possibly seen,” which is safe (it only costs extra exact lookups) but erodes the fast path. Rotating or scaling the filter on a schedule tied to the window horizon restores the pre-filter’s value without ever weakening the exact guarantee.

Because the exact store is the only authority that suppresses, none of these degradations can drop a genuine controlled-substance dispense — the worst case is extra exact lookups or a bounded deferred backlog, both of which preserve the count.

Downstream Integration

The deduplication gate sits between validation and classification in the scan pipeline, and its output feeds the rest of the synchronization backbone as a de-noised, exactly-once signal:

1. Routing & partitioning. Deduplicated events flow into the DEA-tier partitioning of Barcode Scan Log Routing Logic, which assigns each unique scan to its regulatory-priority queue. Deduplication upstream is what lets that router treat every event it receives as a real movement.
2. Distributed exact claims. The authoritative first-writer-wins backend is implemented in Idempotent Scan Ingestion with Redis SETNX, whose atomic SET … NX PX is the concrete ExactStore.claim used above.
3. Probabilistic pre-filtering. The constant-memory membership filter that guards the exact store under peak load is built in Deduplicating Barcode Scans with Bloom Filters, including the false-positive-rate math and the never-drop-on-a-bloom-hit rule.

By collapsing a noisy scan stream into exactly one committed event per physical movement — and auditing every suppression rather than discarding it — the deduplication layer keeps the perpetual inventory count faithful to physical reality and keeps the diversion-alert stream free of phantom shortages.

Frequently Asked Questions

Related

• Up: Data Ingestion & Inventory Sync Workflows — the parent architecture this deduplication layer operates within.
• Deduplicating Barcode Scans with Bloom Filters — the constant-memory probabilistic pre-filter and its false-positive math.
• Idempotent Scan Ingestion with Redis SETNX — the exact, distributed first-writer-wins claim behind the gate.
• Barcode Scan Log Routing Logic — the routing subsystem that consumes de-noised scans.
• Fallback Routing for Offline Sync — deferred commitment and idempotency-key preservation during a disconnect.