Fallback Routing for Offline Sync

Pharmacy inventory systems operate in a zero-tolerance environment for unlogged dispensing events. Network degradation, ISP outages, or POS hardware failures cannot interrupt perpetual inventory tracking or fracture the chain of custody for DEA-controlled substances. The fallback routing workflow detailed here establishes a deterministic, locally-buffered synchronization path that preserves recordkeeping accuracy during connectivity loss and reconciles cleanly once the link returns. It operates within the broader Core Architecture & DEA Compliance Frameworks and guarantees that offline transaction routing never compromises audit integrity, batch traceability, or diversion-detection thresholds.

Regulatory Context & Compliance Boundaries

A disconnected pharmacy workstation is still a registrant under the Controlled Substances Act. The recordkeeping obligation does not pause when the network does, which is why the boundary conditions for offline routing must be encoded as structural constraints rather than handled as a degraded “best effort” mode. Four clauses govern this subsystem directly:

• 21 CFR § 1304.04 — Records of controlled substances must be complete, accurate, and maintained contemporaneously with the transaction. An event captured offline must carry its true point-of-capture timestamp, not the later sync timestamp, or the record is non-contemporaneous on its face.
• 21 CFR § 1304.21 — Every Schedule II–V transaction requires exact product identification. Offline capture cannot defer NDC resolution or schedule classification to reconciliation time, because an unresolved record is an incomplete record.
• HIPAA § 164.312(a)(2)(iv) — ePHI buffered at rest on the edge workstation must be encrypted. A local SQLite queue holding dispensing detail is ePHI at rest the moment it is written.
• FDA 21 CFR § 211.188 — Batch production and control records require lot and expiration traceability, which the buffer must validate at the point of interception rather than at upload.

The architectural rule that follows from these clauses: the local buffer must be append-only, encrypted, fully validated at write time, and cryptographically ordered so that the offline period is indistinguishable from the online period in the final ledger. Everything below enforces that rule.

Routing State Specification

Offline routing is a three-state machine. The system must never silently sit between states, and every transition must be observable in the audit log.

The window-based threshold (rather than a single failed request) prevents transient packet loss from forcing a needless state flap, while still guaranteeing that a genuine outage is captured before a second dispensing event can be lost.

Deterministic Routing Workflow

Each dispensing, receiving, or adjustment event moves through the same ordered path regardless of connectivity state. The only branch is whether the committed record is streamed upstream immediately or held in the buffer.

1. Probe & classify connectivity. An asynchronous health probe targets the primary EDI ingestion endpoint. Three failures inside the 15-second window transition the machine to OFFLINE_ROUTING.
2. Intercept the event. Outbound inventory events are captured by local middleware before they reach the network stack, so no event is ever “in flight and lost.”
3. Normalize and validate. The NDC is normalized to its canonical 11-digit form using the rules defined in NDC-11 vs NDC-10 Parsing Standards, the DEA schedule is resolved against the authoritative DEA Schedule II-V Classification Mapping, and lot/expiration metadata is checked. Invalid payloads are diverted to a quarantine queue.
4. Encrypt and commit. The validated payload is serialized, encrypted with AES-256-GCM, and appended to the buffer with a monotonically increasing sequence ID and a hash-chain pointer to the previous record.
5. Weight by priority. Schedule II events are flagged for immediate reconciliation on restoration; Schedule III–V and OTC events follow FIFO ordering.
6. Reconcile idempotently. When the probe recovers, buffered transactions upload with their original timestamps and an idempotency key, resuming from the last confirmed sequence ID without duplicating records.

Step 1 — Network State Detection & Threshold Trigger

The pharmacy management system (PMS) must monitor upstream connectivity without adding latency to dispensing. A lightweight asynchronous probe issues HTTPS HEAD requests against the ingestion endpoint; when consecutive failures cross the threshold, the routing state machine flips to OFFLINE_ROUTING and middleware begins intercepting events.

import asyncio
import aiohttp
from enum import Enum

class SyncState(Enum):
    ONLINE = "ONLINE"
    OFFLINE_ROUTING = "OFFLINE_ROUTING"
    RECONCILING = "RECONCILING"

class NetworkMonitor:
    def __init__(self, endpoint: str, threshold: int = 3, window_sec: int = 15):
        self.endpoint = endpoint
        self.threshold = threshold
        self.window_sec = window_sec
        self.state = SyncState.ONLINE
        self._failure_count = 0
        self._last_failure_ts = 0.0

    async def probe(self) -> None:
        try:
            async with aiohttp.ClientSession() as session:
                async with session.head(self.endpoint, timeout=3.0) as resp:
                    if resp.status == 200:
                        self._reset_failures()
                        return
        except (aiohttp.ClientError, asyncio.TimeoutError):
            self._record_failure()

    def _record_failure(self) -> None:
        import time
        now = time.time()
        if now - self._last_failure_ts > self.window_sec:
            self._failure_count = 0
        self._failure_count += 1
        self._last_failure_ts = now
        if self._failure_count >= self.threshold:
            self.state = SyncState.OFFLINE_ROUTING

    def _reset_failures(self) -> None:
        self._failure_count = 0
        self.state = SyncState.ONLINE

Step 2 — Local Buffer Initialization & Cryptographic Enforcement

On transition to OFFLINE_ROUTING, a secure encrypted buffer is instantiated on the workstation or edge server. To satisfy § 164.312(a)(2)(iv) while preserving write throughput, the buffer uses SQLite with Write-Ahead Logging (WAL) plus application-level AES-256-GCM payload encryption. The schema enforces strict typing and append-only semantics so records cannot be retroactively altered or deleted — the same immutability guarantee described in Defining Audit Boundaries for Controlled Substances.

import sqlite3
import os
from cryptography.hazmat.primitives.ciphers.aead import AESGCM

class SecureBuffer:
    def __init__(self, db_path: str, master_key: bytes):
        self.db_path = db_path
        self.aesgcm = AESGCM(master_key)
        self._init_db()

    def _init_db(self) -> None:
        os.makedirs(os.path.dirname(self.db_path), exist_ok=True)
        with sqlite3.connect(self.db_path) as conn:
            conn.execute("PRAGMA journal_mode=WAL;")
            conn.execute("""
                CREATE TABLE IF NOT EXISTS offline_txns (
                    seq_id INTEGER PRIMARY KEY AUTOINCREMENT,
                    encrypted_payload BLOB NOT NULL,
                    nonce BLOB NOT NULL,
                    hash_chain_prev TEXT NOT NULL,
                    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
                ) STRICT;
            """)

    def insert_encrypted(self, payload: bytes, prev_hash: str) -> int:
        nonce = os.urandom(12)
        ciphertext = self.aesgcm.encrypt(nonce, payload, None)
        with sqlite3.connect(self.db_path) as conn:
            cursor = conn.execute(
                "INSERT INTO offline_txns (encrypted_payload, nonce, hash_chain_prev) VALUES (?, ?, ?)",
                (ciphertext, nonce, prev_hash)
            )
            return cursor.lastrowid

Step 3 — Transaction Interception & Schema Validation

Every event is serialized and validated against the master drug file before it can enter the buffer. NDC formatting is normalized to the canonical 11-digit representation to prevent reconciliation mismatches, schedule classification is resolved concurrently, and lot/expiration metadata is verified. A payload that fails any check is rejected — never silently dropped — so the offline record set stays complete and § 1304.21-compliant.

import json
import re
from pydantic import BaseModel, field_validator
from typing import Optional

class InventoryEvent(BaseModel):
    ndc: str
    schedule: Optional[str] = None
    lot_number: str
    expiration_date: str
    quantity_dispensed: int
    timestamp: str

    @field_validator("ndc")
    @classmethod
    def normalize_ndc(cls, v: str) -> str:
        digits = re.sub(r"\D", "", v)
        if len(digits) == 10:
            return f"0{digits[:5]}-{digits[5:7]}-{digits[7:]}"
        elif len(digits) == 11:
            return f"{digits[:5]}-{digits[5:7]}-{digits[7:]}"
        raise ValueError("NDC must be 10 or 11 digits")

    def to_payload(self) -> bytes:
        return json.dumps(self.model_dump(mode="json")).encode("utf-8")

Step 4 — Fallback Routing Execution & Priority Weighting

Validated transactions are written to the buffer with a monotonically increasing sequence ID and a cryptographic hash-chain pointer. The router applies deterministic priority weighting so Schedule II events are reconciled first on restoration. The concrete edge-side queue mechanics — overflow thresholds, circuit-breaker behavior, and TCP keepalive tuning — are detailed in the companion guide on fallback sync architecture for disconnected POS systems.

import hashlib
from dataclasses import dataclass
from typing import List

# Defined elsewhere on this page (see the surrounding blocks):
# - InventoryEvent
# - SecureBuffer

@dataclass
class PriorityTransaction:
    seq_id: int
    payload_hash: str
    schedule: str
    raw_payload: bytes

class FallbackRouter:
    def __init__(self, buffer: SecureBuffer):
        self.buffer = buffer
        self.prev_hash = "0" * 64  # Genesis hash
        self.priority_queue: List[PriorityTransaction] = []

    def route_transaction(self, event: InventoryEvent) -> None:
        payload = event.to_payload()
        current_hash = hashlib.sha256(payload).hexdigest()
        chain_hash = hashlib.sha256(f"{self.prev_hash}{current_hash}".encode()).hexdigest()

        seq_id = self.buffer.insert_encrypted(payload, self.prev_hash)
        self.prev_hash = chain_hash

        self.priority_queue.append(PriorityTransaction(
            seq_id=seq_id,
            payload_hash=current_hash,
            schedule=event.schedule or "OTC",
            raw_payload=payload
        ))
        self._sort_by_priority()

    def _sort_by_priority(self) -> None:
        # Schedule II first, then III-V, then OTC
        schedule_order = {"II": 0, "III": 1, "IV": 2, "V": 3, "OTC": 4}
        self.priority_queue.sort(key=lambda x: schedule_order.get(x.schedule, 5))

Step 5 — Connectivity Restoration & Idempotent Reconciliation

When the probe recovers, the daemon initiates a batched, idempotent upload. Each transaction carries its original timestamp, cryptographic nonce, and hash-chain pointer to guarantee exactly-once delivery. Server acknowledgments are validated against local sequence IDs; a mid-batch failure resumes from the last confirmed seq_id without duplication. The retry and back-off behavior here mirrors the patterns in the data-ingestion error handling & retry mechanisms workflow.

import httpx
from typing import Dict, Any

# Defined elsewhere on this page (see the surrounding blocks):
# - FallbackRouter

class ReconciliationDaemon:
    def __init__(self, router: FallbackRouter, sync_endpoint: str):
        self.router = router
        self.sync_endpoint = sync_endpoint
        self.last_confirmed_seq = 0

    async def reconcile(self) -> Dict[str, Any]:
        if not self.router.priority_queue:
            return {"status": "idle"}

        batch = self.router.priority_queue[self.last_confirmed_seq:]
        results = {"uploaded": 0, "failed": []}

        async with httpx.AsyncClient(timeout=10.0) as client:
            for txn in batch:
                headers = {
                    "X-Idempotency-Key": txn.payload_hash,
                    "X-Sequence-ID": str(txn.seq_id),
                    "Content-Type": "application/json"
                }
                try:
                    resp = await client.post(
                        self.sync_endpoint,
                        content=txn.raw_payload,
                        headers=headers
                    )
                    if resp.status_code in (200, 201, 202):
                        results["uploaded"] += 1
                        self.last_confirmed_seq = txn.seq_id
                    elif resp.status_code == 409:
                        # Already processed by server
                        self.last_confirmed_seq = txn.seq_id
                        results["uploaded"] += 1
                    else:
                        results["failed"].append(txn.seq_id)
                except httpx.RequestError:
                    results["failed"].append(txn.seq_id)
                    break  # Halt on network failure to preserve order

        return results

Compliance Mapping & Audit Boundaries

The fallback routing layer satisfies explicit regulatory controls without manual intervention during connectivity loss. Each buffered record feeds the same append-only ledger that the online path uses, so the audit boundary is identical in both states.

Access to the buffer and reconciliation daemon must be governed by the same role-based access controls and access-log requirements that protect the central ledger, established in the Pharmacy Security Framework Architecture. No offline path may widen the privilege surface that the online path enforces.

Error Handling & Offline Resilience

Resilience is defined here as the guarantee that no event is lost, duplicated, or silently mutated regardless of how the failure manifests.

• Quarantine queue. Payloads that fail NDC normalization, schedule resolution, or lot/expiration validation are written to a REJECTED_QUEUE with an explicit error code rather than dropped. A pharmacist resolves them on restoration; the count of quarantined records is surfaced as an operational alert so a validation defect cannot quietly suppress dispensing records.
• Idempotency-key collision safety. The idempotency key is the SHA-256 of the canonical payload. Two genuinely distinct events (different lot, quantity, or timestamp) produce different hashes; a 409 response confirms the server already holds that exact record and is treated as success, not error.
• Order preservation under partial failure. Reconciliation halts on the first network error rather than skipping ahead, so the hash chain is never uploaded out of order. The next pass resumes from the last confirmed sequence ID.
• Buffer overflow. When local storage crosses its configured ceiling, the circuit breaker described in the disconnected-POS guide throttles non-controlled OTC capture before it ever throttles Schedule II–V capture, keeping the highest-risk records flowing into the buffer.

Downstream Integration

Once reconciled, buffered records are indistinguishable from online events and flow into the same downstream subsystems. The reconciliation daemon’s output is consumed by the async batch processing for inventory updates pipeline, which posts decrements and receipts to the perpetual inventory. Because each record retains its original capture timestamp, diversion-detection algorithms operate on accurate temporal data even for events captured during an outage, and the broader Data Ingestion & Inventory Sync Workflows layer treats reconciled and live records under one validation contract.

Frequently Asked Questions

Related

• Core Architecture & DEA Compliance Frameworks — parent framework for this routing pattern
• Fallback sync architecture for disconnected POS systems — edge-side queue, keepalive, and circuit-breaker detail
• DEA Schedule II-V Classification Mapping — schedule resolution feeding priority weighting
• NDC-11 vs NDC-10 Parsing Standards — normalization rules applied at interception
• Error handling & retry mechanisms — back-off patterns reused during reconciliation

For implementation references, see the Python Cryptography Documentation, the NIST SP 800-111 Guide to Storage Security, and the controlled-substance recordkeeping text in 21 CFR Part 1304.