API Polling & Webhook Integration for DSCSA Serialization Compliance

Part of the Serialization Data Ingestion & EPCIS Event Sync pipeline, this guide covers the transport layer where serialized events actually cross the boundary between trading partners. Pharmaceutical track-and-trace under the Drug Supply Chain Security Act (DSCSA) demands deterministic, auditable, and near-real-time exchange of EPCIS events, Transaction Information (TI), Transaction History (TH), and Transaction Statements (TS) between manufacturers, contract manufacturers (CMOs), wholesale distributors, and verification infrastructure. Legacy EDI and nightly batch-file transfers cannot express unit-level traceability at the latency modern recall and verification workflows require. The specific problem this page solves is choosing and implementing the correct ingestion mechanism — outbound polling, inbound webhooks, or the hybrid of both — so that every serialized identifier lands in your EPCIS repository exactly once, in order, and with a verifiable chain of custody.

Architecture Diagram #

Two ingestion vectors converge on a single idempotent merge. Polling pulls events from partners that cannot push; webhooks receive events the moment a partner commissions, aggregates, ships, or receives product. Both paths write through the same deduplication and merge stage so the EPCIS repository never sees a duplicate or an out-of-order commit.

Figure — Polling and webhook ingestion converging on an idempotent merge.

Foundational Concepts & Data Contracts #

Before writing transport code, fix the data contracts that both vectors must honor. Every ingested payload is a GS1 EPCIS 1.2 or 2.0 document carrying one or more event types that must appear verbatim in your schemas: ObjectEvent for commissioning and observation, AggregationEvent for parent-child packaging, TransactionEvent for change of ownership, and TransformationEvent for repackaging. Each event references saleable units by their Serialized GTIN — the 14-digit GTIN under Application Identifier (01) paired with the serial number under (21) — plus the expiration date (17) and lot (10). Logistics units are keyed by SSCC (00), and every read point and trading-partner party resolves to a GLN. The identifier-and-event correctness rules behind these fields are defined in GS1 Standards Implementation; the ingestion layer’s job is to move those documents intact, not to reinterpret them.

Three contract-level concepts govern the transport itself:

• Cursor / high-water mark. Polling is stateless on the partner side, so your client owns the position it has consumed. A monotonic cursor — an eventTime watermark or an opaque nextPageToken returned by the partner — must be persisted transactionally with the events it covers, or a crash will silently skip or replay a window.
• Idempotency key. Every event needs a stable identity independent of transport. For EPCIS this is the eventID (an EPCIS 2.0 URN or UUID); where a partner omits it, derive a deterministic hash over the canonical event body. Both vectors deduplicate on this key so a webhook retry and a polling backfill of the same event collapse to one row.
• Delivery semantics. Networks give you at-least-once delivery at best. The merge stage upgrades that to effectively-once by rejecting any event whose idempotency key already exists, which is the only way to keep TH reconstruction accurate under DSCSA.

Polling vs. webhooks in regulated environments #

The choice is rarely binary. Most enterprise serialization platforms run a hybrid topology to accommodate heterogeneous partner capabilities, legacy ERP/WMS egress constraints, and FDA Verification Router Service requirements.

Polling remains the standard for deterministic retrieval from systems that lack outbound event capability or enforce strict firewall egress rules. It gives predictable retry semantics, explicit rate control, and straightforward audit logging, but it introduces latency, burns compute during idle windows, and needs careful backoff so it does not trip a partner’s anti-abuse throttles. Webhooks invert every one of those properties: sub-second latency and no wasted calls, at the cost of a receiver that must verify cryptographic signatures, enforce idempotency, and degrade gracefully under payload bursts, malformed JSON, or transient downstream failures. In a DSCSA-compliant pipeline, polling typically handles periodic reconciliation, VRS status checks, and historical backfills, while webhooks drive real-time ingestion and immediate compliance flagging.

Step-by-Step Implementation #

The following steps build the ingestion layer as production Python (3.10+). Each step names the DSCSA or GS1 rule it satisfies.

Step 1 — Build a resilient polling client with explicit retry semantics #

Polling against partner and FDA-adjacent endpoints must be traceable and must never hammer an endpoint into a rate-limit ban. Bounded exponential backoff with a status_forcelist satisfies the availability and recordkeeping expectations of DSCSA data exchange while respecting partner SLAs.

import logging
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

logger = logging.getLogger("dscsa.polling_client")


class DSCSAPollingClient:
    def __init__(self, base_url: str, api_key: str, timeout: float = 15.0):
        self.base_url = base_url.rstrip("/")
        self.session = requests.Session()
        self.session.headers.update(
            {"Authorization": f"Bearer {api_key}", "Accept": "application/json"}
        )
        self.timeout = timeout

        retry_strategy = Retry(
            total=5,
            backoff_factor=1.0,
            status_forcelist=[429, 500, 502, 503, 504],
            allowed_methods=["GET", "POST"],
            respect_retry_after_header=True,
        )
        adapter = HTTPAdapter(max_retries=retry_strategy)
        self.session.mount("https://", adapter)
        self.session.mount("http://", adapter)

    def fetch_epcis_events(self, endpoint: str, params: dict) -> dict:
        url = f"{self.base_url}/{endpoint}"
        try:
            response = self.session.get(url, params=params, timeout=self.timeout)
            response.raise_for_status()
            logger.info("Retrieved EPCIS payload from %s", url)
            return response.json()
        except requests.exceptions.RequestException as exc:
            logger.error("Polling request failed for %s: %s", url, exc)
            raise

This client feeds the async batch processing pipelines that normalize, deduplicate, and persist retrieved payloads. When polling the FDA Verification Router Service or a high-traffic partner API, strict adherence to the patterns in handling rate limits on FDA verification APIs is mandatory to stay within SLA.

Step 2 — Advance a durable cursor so polling never skips or replays #

DSCSA requires an accurate, gap-free TH. A poll must persist its new cursor in the same transaction that persists the events it fetched, so a mid-batch crash resumes from the last committed position rather than losing a window of serialized events.

from datetime import datetime, timezone


def poll_since_cursor(client: DSCSAPollingClient, endpoint: str, cursor: str) -> tuple[list[dict], str]:
    """Fetch one page of events strictly newer than the stored cursor.

    Returns the events plus the next cursor. The caller must persist BOTH
    atomically so an interrupted run resumes exactly where it stopped.
    """
    params = {"GE_eventTime": cursor, "orderBy": "eventTime", "perPage": 500}
    body = client.fetch_epcis_events(endpoint, params)
    events = body.get("epcisBody", {}).get("eventList", [])

    if not events:
        return [], cursor

    # Cursor is the max eventTime seen; ISO 8601 with explicit offset per EPCIS.
    next_cursor = max(e["eventTime"] for e in events)
    logger.info("Polled %d events; cursor %s -> %s", len(events), cursor, next_cursor)
    return events, next_cursor

Step 3 — Verify inbound webhook signatures before any processing #

Webhook payloads are untrusted until proven authentic. Partners sign the raw request body with HMAC-SHA256 over a shared secret; the receiver recomputes the digest and compares it in constant time to defeat timing attacks. Rejecting unsigned or mismatched payloads at the edge is what keeps a forged event out of your TH.

import hashlib
import hmac
from fastapi import Request, HTTPException


async def verify_webhook_signature(request: Request, payload: bytes, shared_secret: str) -> bool:
    signature_header = request.headers.get("X-Signature", "")
    expected = hmac.new(
        shared_secret.encode("utf-8"),
        payload,
        hashlib.sha256,
    ).hexdigest()
    if not hmac.compare_digest(signature_header, expected):
        raise HTTPException(status_code=401, detail="Invalid webhook signature")
    return True

Figure — The inbound-webhook state machine: a linear happy path with a distinct terminal response for each way a payload can fall out.

Step 4 — Enforce idempotency and replay protection at the merge boundary #

DSCSA mandates accurate TH reconstruction, so a duplicate delivery — common during network partitions and partner retries — must never create a second EPCIS event or corrupt inventory state. A TTL-bounded idempotency cache keyed to the event’s stable identity turns at-least-once delivery into effectively-once processing.

import redis

_cache = redis.Redis(host="localhost", port=6379, db=0)
_TTL_SECONDS = 60 * 60 * 24 * 7  # replay window: one week


def claim_event(idempotency_key: str) -> bool:
    """Return True the first time a key is seen, False on any replay.

    SET NX is atomic, so concurrent webhook and polling deliveries of the
    same event race to exactly one winner.
    """
    return bool(_cache.set(idempotency_key, "1", nx=True, ex=_TTL_SECONDS))

Only events that win the claim_event race proceed to schema validation and persistence; losers are acknowledged with a 200 no-op so the partner stops retrying. Both the polling path from Step 2 and the webhook path from Step 3 funnel through this single gate, which is what lets the two vectors coexist without double-counting a shipment.

Validation & Error Handling #

Authenticated does not mean well-formed. Before business logic runs, each event is validated against strict EPCIS schema definitions so a malformed payload is caught, quarantined, and reported without halting the pipeline. Structural defects — a missing eventTime, a serial that fails its check digit, an AggregationEvent with no parent — are routed to a dead-letter queue (DLQ) with the original payload, a structured error code, and full request context for manual compliance review; the deep patterns for this stage live in Schema Validation & Error Handling. Transient failures such as a downstream 503 are retried with backoff. The rule of thumb: never drop an event silently and never let one bad event block the ones behind it. When a quarantined event turns out to indicate a genuine anomaly rather than a formatting bug, it should escalate into Suspect Product Investigation Workflows with its payload hash attached as evidence.

For the webhook receiver specifically, size limits and content-type checks belong ahead of JSON parsing: reject bodies over a hard ceiling with 413 and non-JSON content with 415 so a hostile or misconfigured partner cannot exhaust memory. A payload that passes transport checks but fails schema validation returns 202 Accepted with the event routed to the DLQ, distinguishing “we have it and will review it” from “resend this.”

Performance & Scalability Considerations #

At line speeds of tens of thousands of units per hour, the ingestion layer is throughput-bound, not CPU-bound. Poll pages should be sized to the partner’s perPage ceiling (commonly 500–1000 events) and issued with bounded concurrency — a semaphore capping simultaneous partner requests prevents your own scheduler from becoming the source of a rate-limit storm. Webhook receivers should acknowledge fast and defer work: verify the signature and idempotency key inline, then hand the raw event to a broker (Kafka or RabbitMQ) so the HTTP response returns in milliseconds even when the downstream EPCIS write is momentarily slow. This decoupling also absorbs bursts, which is essential when a partner flushes a full pallet’s worth of aggregation events in a single second.

Tune three levers. Broker partitioning by SGTIN or SSCC preserves per-unit event ordering while allowing parallel consumers. The idempotency cache TTL must exceed the longest realistic partner retry window, or a late replay slips through as a duplicate. And polling frequency should be adaptive: back off toward the reconciliation cadence during quiet periods and tighten it during active shipping windows, letting webhooks carry the real-time load. Sustained high-volume streams are better served by the real-time event stream processing topology, with polling reserved for reconciliation and backfill.

Audit & Compliance Checkpoints #

Regardless of the ingestion vector, DSCSA compliance rests on data integrity, retention, and traceability. The ingestion layer must produce inspector-facing evidence for each of these:

• Immutable audit logging. Every API request, webhook receipt, signature-verification result, idempotency decision, and schema-validation outcome is written to an append-only log capturing the UTC timestamp, correlation ID, source IP, and payload hash. This is the substrate for 21 CFR Part 11 electronic-record requirements.
• State reconciliation. Polling and webhooks will diverge during partner outages or maintenance. A daily reconciliation job compares webhook-received event counts against a polling-retrieved master dataset per partner and per SGTIN range, surfacing silent data loss before it becomes an audit finding.
• Retention and purging. Serialized data and associated EPCIS events must be retained for a minimum of six years under DSCSA and remain reproducible in their original structure. Lifecycle policies archive cold data to compliant object storage while keeping queryable indexes for rapid recall execution.
• Cryptographic provenance. Store the received HMAC signature and the computed digest alongside each webhook event so an inspector can later confirm the event was authentically delivered by the named trading partner.

Troubleshooting #

Frequently Asked Questions #

When should I poll and when should I use webhooks? Use webhooks for partners that can push and for anything needing sub-second latency — commissioning, shipping, and receiving events that drive real-time compliance flags. Use polling for partners that cannot emit outbound events, for VRS status checks, and for periodic reconciliation and historical backfills. Most production pipelines run both.

How do I stop duplicate webhook deliveries from corrupting Transaction History? Deduplicate on a stable idempotency key — the EPCIS eventID or a deterministic hash of the canonical event body — using an atomic SET NX claim with a TTL longer than the partner’s retry window. Only the first delivery is processed; replays return a 200 no-op.

Why verify the HMAC signature over the raw body instead of the parsed JSON? Re-serializing parsed JSON can reorder keys or change whitespace, producing a different digest than the partner signed. Always compute HMAC-SHA256 over the exact bytes received, then compare in constant time with hmac.compare_digest to avoid timing attacks.

What happens to a payload that is authentic but fails schema validation? It is routed to a dead-letter queue with a structured error code and the original payload for manual review, and the receiver returns 202 Accepted — never dropped silently and never allowed to block valid events behind it. Genuine anomalies escalate into a suspect-product investigation.

How long must ingested EPCIS event data be retained? Transaction Information and Transaction Statements must be retained for six years under DSCSA and remain reproducible in their original EPCIS structure, backed by an immutable, hash-indexed audit log to satisfy 21 CFR Part 11.

Conclusion #

Polling and webhooks are complementary mechanisms in a DSCSA-compliant serialization architecture, not competing ones. Polling delivers deterministic control for reconciliation, legacy-partner integration, and regulatory backfills; webhooks deliver the low-latency event streams modern supply-chain visibility depends on. By funneling both vectors through a single idempotent merge — with HMAC verification, durable cursors, strict schema validation, exponential backoff, and immutable audit trails — engineering teams build an ingestion layer that satisfies FDA expectations while scaling to enterprise serialization volumes. The result is that every unit-level transaction remains verifiable, traceable, and audit-ready from commissioning to dispensing.

Related #

• Serialization Data Ingestion & EPCIS Event Sync — the parent pipeline this ingestion layer belongs to.
• Handling rate limits on FDA verification APIs — backoff and quota strategy for VRS and partner endpoints.
• Async Batch Processing Pipelines — where polled payloads are normalized and persisted.
• Real-Time Event Stream Processing — the streaming topology for sustained high-volume webhook loads.
• Schema Validation & Error Handling — quarantine and dead-letter patterns for malformed events.