Real-Time Monitoring & Alerting

Real-time monitoring and alerting is the operational nervous system of Serialization Data Ingestion & EPCIS Event Sync, the section this topic is part of, and it exists to answer one uncomfortable question before an FDA inspector or trading partner does: is the pipeline actually keeping up right now? Under the Drug Supply Chain Security Act (DSCSA), a serialization ingestion pipeline that silently stalls does not throw a visible error — it simply stops committing EPCIS records. Packaging lines keep printing serialized (01)/(21) DataMatrix codes, trucks keep leaving the dock, but the electronic Transaction Information falls further and further behind the physical product. Hours later, a partner sends a verification request for a saleable unit that was shipped this morning, and the repository returns not found for a unit that is perfectly legitimate. The stall was never loud; it was silent, and silence in a compliance pipeline is the most expensive failure mode there is. This topic covers the metrics, dashboards, threshold alerting, on-call runbooks, and dead-letter-queue drain workflows that convert that silence into a page before the missing-record window becomes an audit finding.

Architecture #

Observability for a serialization ingestion pipeline is a second, parallel data path. The primary path moves scans through validation into the EPCIS repository; the observability path samples that primary path continuously, exports numeric time series, and evaluates them against service-level objectives that map back to DSCSA obligations. The reference topology below keeps instrumentation, collection, visualization, and alerting as decoupled stages, so a busy dashboard query can never slow down the ingest workers and an alerting outage can never block event commits.

Figure — observability path for a serialization ingestion pipeline.

Three design choices make this topology defensible. First, the exporter exposes a pull-based /metrics endpoint rather than pushing, so Prometheus scrape failures are themselves a signal — a target that stops responding is an alert, not a blind spot. Second, dashboards and the alert evaluator read the same time-series database but run independently, so an analyst zooming into a week of history never competes with the alerting loop for resources. Third, every alert links directly to a runbook, because a page that tells an on-call engineer what broke but not what to do next wastes the most expensive minutes of an incident. The same event streams that feed these metrics are governed upstream by real-time event stream processing, and the malformed events that inflate the dead-letter queue originate in the schema validation and error handling layer — monitoring is where both become visible.

Foundational Concepts & Data Contracts #

Before you can alert on a serialization pipeline you have to agree on what you are measuring, and in a DSCSA context the measurements are not generic infrastructure metrics — they are compliance signals with regulatory meaning. Four metrics carry that meaning.

Ingestion lag is the wall-clock age of the oldest event that has been produced but not yet committed to the EPCIS repository. It is measured in seconds and is the single most important number on the board, because it is the width of the window during which a shipped unit cannot be verified. When lag exceeds the time it takes product to reach a downstream partner, you have a live compliance exposure.

Dead-letter-queue depth is the count of events that failed validation and were quarantined rather than committed. A flat, low DLQ depth is healthy; a rising slope means an upstream data-quality regression — a line changeover that printed a malformed GTIN (01), or a partner that started emitting an unexpected ObjectEvent shape. Depth is a leading indicator that something will breach if left unattended.

Verification latency is the round-trip time for a verification lookup, tracked as a distribution (not just an average) so you can alert on the p95 or p99 tail. DSCSA verification responses are time-sensitive, and a fat tail means some fraction of partner queries are timing out even when the median looks fine.

The commission-versus-verify gap counts serials that were commissioned upstream but have not yet been observed being verified downstream, bucketed by GTIN. A persistent, growing gap is the fingerprint of a silent stall: events are entering the pipeline but not completing their journey into a queryable, verifiable state.

These metrics attach to a small set of stable identifiers. Every counter is labelled by the trading-partner GLN so a single partner’s regression is isolable, and gap and latency gauges are labelled by the 14-digit GTIN so you can trace a spike to a specific product. EPCIS event type names — ObjectEvent, AggregationEvent, TransactionEvent, TransformationEvent — become label values, so a board can show that aggregation events are lagging while object events are current. The data contract for the metrics layer is therefore: numeric time series, low-cardinality labels (GLN, GTIN root, event type, DLQ reason), and monotonic counters for anything you will later compute a rate over. Get the label cardinality wrong — labelling by full serial (21), for instance — and the time-series database itself becomes the outage.

Step-by-Step Implementation #

The implementation below builds the observability path in five stages: define the metric contract, instrument the ingest worker, sample the derived gauges on an interval, evaluate SLA rules asynchronously, and expose a controlled drain path for the dead-letter queue. Each stage names the DSCSA or GS1 obligation it protects.

Step 1 — Define the metric contract with the Prometheus client #

Declare every metric once, at module scope, so the exporter presents a stable contract Prometheus can scrape. Counters are monotonic; gauges are point-in-time samples; the histogram captures the latency distribution with buckets tuned to a verification SLA.

from prometheus_client import Counter, Gauge, Histogram

# Throughput and failures, labelled by trading partner and event type.
EVENTS_INGESTED = Counter(
    "epcis_events_ingested_total",
    "EPCIS events committed to the repository.",
    ["partner_gln", "event_type"],
)
EVENTS_DEADLETTERED = Counter(
    "epcis_events_deadlettered_total",
    "Events routed to the dead-letter queue.",
    ["partner_gln", "reason"],
)

# Point-in-time depths and gaps sampled by a background collector.
DLQ_DEPTH = Gauge(
    "epcis_dlq_depth",
    "Current dead-letter-queue backlog, in messages.",
    ["queue"],
)
INGESTION_LAG = Gauge(
    "epcis_ingestion_lag_seconds",
    "Wall-clock age of the oldest un-committed event.",
    ["partner_gln"],
)
COMMISSION_VERIFY_GAP = Gauge(
    "epcis_commission_verify_gap",
    "Serials commissioned upstream but not yet verified downstream.",
    ["gtin"],
)

# Latency distribution for verification round-trips.
VERIFY_LATENCY = Histogram(
    "epcis_verify_latency_seconds",
    "Verification request round-trip latency.",
    ["partner_gln"],
    buckets=(0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0),
)

This satisfies the DSCSA expectation that a trading partner can be answered promptly: labelling by partner_gln means an SLA breach can be attributed to one relationship instead of averaged into invisibility, and the histogram buckets make the response-time tail observable rather than hidden behind a mean.

Step 2 — Instrument the ingest worker without blocking it #

Record metrics inline as events are processed, but keep the instrumentation cheap: incrementing a counter and setting a gauge are non-blocking, in-process operations. The worker still commits valid events and quarantines malformed ones exactly as the validation layer dictates.

import time
from prometheus_client import start_http_server
from pydantic import ValidationError

# SerializedUnit is the typed contract from the schema-validation layer.


def start_metrics_endpoint(port: int = 9108) -> None:
    """Expose /metrics so Prometheus can scrape this worker."""
    start_http_server(port)


async def ingest_one(raw, epcis_repo, dlq) -> None:
    partner = raw.headers.get("partner_gln", "unknown")
    try:
        unit = SerializedUnit.model_validate(raw.payload)
    except ValidationError as err:
        EVENTS_DEADLETTERED.labels(partner_gln=partner, reason="schema").inc()
        await dlq.put({"payload": raw.payload, "errors": err.errors()})
        return
    await epcis_repo.commit_object_event(unit)
    EVENTS_INGESTED.labels(partner_gln=partner, event_type="ObjectEvent").inc()
    INGESTION_LAG.labels(partner_gln=partner).set(time.time() - raw.produced_at)

This satisfies the DSCSA requirement that the packaging line never halts for a compliance service: metrics recording adds microseconds, and a validation failure increments a dead-letter counter instead of raising, so a single bad (01)/(21) payload is counted and quarantined rather than allowed to stop the stream.

Step 3 — Sample the commission-versus-verify gap on an interval #

The gap and the per-partner lag cannot be derived from a single event — they are aggregate queries against the repository. A background task samples them on a fixed interval and writes the results into gauges, so the expensive query runs on a predictable schedule instead of on every request.

import asyncio


async def refresh_derived_gauges(repo, poll_seconds: int = 30) -> None:
    """Sample the commission-vs-verify gap and lag per identifier."""
    while True:
        for row in await repo.fetch_gap_by_gtin():
            COMMISSION_VERIFY_GAP.labels(gtin=row["gtin"]).set(row["gap"])
        for row in await repo.fetch_lag_by_partner():
            INGESTION_LAG.labels(partner_gln=row["partner_gln"]).set(
                row["lag_seconds"]
            )
        await asyncio.sleep(poll_seconds)

This satisfies the DSCSA traceability obligation: a growing commission-versus-verify gap for a GTIN means saleable units exist that the repository cannot yet answer for, and sampling it every 30 seconds surfaces the silent stall long before a partner’s verification request exposes it.

Step 4 — Evaluate SLA rules asynchronously with a sustained-breach window #

A good alert does not fire the instant a metric ticks over a threshold — a one-second spike in lag is noise. It fires when the threshold is breached and stays breached for a sustained window, mirroring Prometheus’s for: clause. The evaluator below tracks when each rule’s breach began and only notifies once the breach has persisted.

import time
from dataclasses import dataclass
from collections.abc import Awaitable, Callable


@dataclass(frozen=True)
class Rule:
    name: str
    metric: Callable[[], float]
    threshold: float
    for_seconds: float          # breach must persist this long
    severity: str
    runbook: str


class AlertEvaluator:
    def __init__(self, rules: list[Rule],
                 notify: Callable[[str, str, str], Awaitable[None]]) -> None:
        self._rules = rules
        self._notify = notify
        self._breach_started: dict[str, float] = {}

    async def evaluate_once(self, now: float) -> None:
        for rule in self._rules:
            if rule.metric() <= rule.threshold:
                self._breach_started.pop(rule.name, None)
                continue
            started = self._breach_started.setdefault(rule.name, now)
            if now - started >= rule.for_seconds:
                await self._notify(
                    rule.severity,
                    f"{rule.name}: {rule.metric():.2f} > {rule.threshold}",
                    rule.runbook,
                )

    async def run(self, sample_seconds: float = 15.0) -> None:
        while True:
            await self.evaluate_once(time.monotonic())
            await asyncio.sleep(sample_seconds)

This satisfies the operational side of DSCSA verification readiness: a sustained-breach window suppresses flapping so on-call engineers trust the pages they get, and every rule carries a runbook link so the page tells the responder what to do. Configuring these thresholds against real service-level objectives is covered in depth in configuring SLA alerts for serialization pipelines.

Step 5 — Expose a controlled dead-letter-queue drain path #

When the DLQ-depth alert fires and the root cause is fixed, the backlog has to be replayed — but blindly re-committing everything risks re-quarantining genuinely bad data in a tight loop. The drain replays in bounded batches, parks anything still invalid, and decrements the depth gauge so the board reflects progress in real time.

async def drain_dead_letters(dlq, epcis_repo, max_batch: int = 200) -> dict:
    """Replay quarantined events after the root cause is resolved."""
    replayed, still_failing = 0, 0
    for _ in range(max_batch):
        item = await dlq.get_nowait()
        if item is None:
            break
        try:
            unit = SerializedUnit.model_validate(item["payload"])
        except ValidationError:
            await dlq.park(item)        # leave genuinely bad data parked
            still_failing += 1
            continue
        await epcis_repo.commit_object_event(unit)
        DLQ_DEPTH.labels(queue="epcis-ingest").dec()
        replayed += 1
    return {"replayed": replayed, "still_failing": still_failing}

This satisfies the six-year DSCSA retention obligation: nothing is deleted during a drain — recoverable events are committed and unrecoverable ones stay parked with their original payload for investigation. The full safety protocol, including poison-message detection and replay ordering, is detailed in draining dead-letter queues safely.

Validation & Error Handling #

Monitoring infrastructure fails in ways that are especially dangerous because a broken monitor looks exactly like a healthy pipeline: no alerts. The discipline is to treat the absence of data as an error state. A Prometheus scrape target that goes silent must itself trigger an alert (a up == 0 rule), because a crashed exporter produces the same empty dashboard as a perfectly idle pipeline. Similarly, a derived gauge that has not been refreshed within two poll intervals is stale, and stale data must be visibly marked stale rather than displayed as if it were current — an operator making a shipping decision off a frozen gap gauge is worse off than one who knows the number is unavailable.

Cardinality is the other failure surface. Every label value creates a distinct time series, so labelling a counter by full serial (21) or by raw error string can spawn millions of series and take down the time-series database — turning your observability system into the outage it was meant to catch. Keep labels bounded: partner GLN, GTIN root, event type, and a small enumerated set of DLQ reasons. Free-form detail belongs in structured logs, not metric labels. Finally, the alert evaluator must be defensive about its own metric reads: a metric() callable that raises because the repository is unreachable should be caught and converted into a distinct evaluator degraded alert, never silently swallowed, so a monitoring failure is never mistaken for an all-clear.

Performance & Scalability #

The observability path must scale sub-linearly relative to ingest throughput, or it becomes a bottleneck at exactly the peak-shipping moments when you most need it. Three levers keep it cheap. First, in-process counter increments are effectively free — the cost is paid at scrape time when Prometheus pulls the exposition text, so tune the scrape interval (typically 15–30 seconds) rather than recording metrics more eagerly. Second, derived gauges that require repository queries — the commission-versus-verify gap especially — must be sampled on an interval and cached, never computed per request; a 30-second poll against an indexed aggregate query costs a fraction of what recomputing it on every dashboard refresh would.

Third, histograms are the expensive metric type because each labelled series carries a bucket array, so choose buckets deliberately and keep the label set small. For a verification-latency histogram, eight buckets across two partners is sixteen series plus counts and sums; the same histogram labelled by GTIN would explode. When ingest volume grows past what a single worker can instrument, run the exporter per worker and let Prometheus aggregate across targets — horizontal scale on the ingest side does not require any change to the metric contract, only more scrape targets. The throughput ceiling of the pipeline being observed is itself governed by the async batch processing pipelines patterns; monitoring simply has to stay light enough that it never becomes the limiting factor.

Audit & Compliance Checkpoints #

Observability data is not decoration — during an inspection it becomes evidence that the traceability system was operating correctly, so parts of it fall under the same retention and integrity rules as the EPCIS records themselves. Three checkpoints matter. First, alert history is an audit artifact: when an ingestion stall occurred, when it was detected, and when it was resolved together demonstrate that a missing-record window was bounded and remediated, which is exactly what a warning-letter response needs to show. Retain fired-alert records and their resolution timestamps alongside the operational logs.

Second, every dead-letter drain is a chain-of-custody event. Replaying quarantined data changes what the repository can answer for, so each drain run should append an immutable, timestamped record — how many events were replayed, how many stayed parked, and the acting identity — to the same hash-chained audit trail that governs the rest of the DSCSA lifecycle under 21 CFR Part 11. Third, the metrics that back an SLA must be retained long enough to prove the SLA was met over the reporting period; the DSCSA transaction-information retention window is six years, and while raw high-resolution metrics are usually downsampled well before then, the compliance-facing rollups (daily p99 verification latency, peak DLQ depth, maximum ingestion lag) should be retained as durable evidence. The practical test is the same one that governs the whole architecture: when an inspector asks whether you could have answered a verification request on a given day, your monitoring history should answer yes, and here is the lag and latency data that proves it.

Troubleshooting #

Building the panels that surface these signals is a topic in its own right — see building EPCIS ingestion dashboards with Python for the query and layout patterns that make a stall obvious at a glance.

Frequently Asked Questions #

Why is a silent ingestion stall a DSCSA compliance problem, not just an ops problem? #

Because the physical product keeps moving while the electronic record falls behind. A stalled pipeline still lets serialized units ship, but their EPCIS events are not yet queryable, so a trading partner’s verification request returns not found for a legitimate unit. That gap between shipped-and-unverifiable is a live compliance exposure, and monitoring exists to bound it to minutes instead of hours.

What is the single most important metric for a serialization ingestion pipeline? #

Ingestion lag — the age of the oldest un-committed event — because it is literally the width of the window during which a shipped unit cannot be verified. Dead-letter-queue depth and the commission-versus-verify gap are close seconds as leading indicators, but lag is the number that most directly maps to a DSCSA verification-readiness failure.

How do I stop alerts from flapping on brief metric spikes? #

Require a sustained breach. Instead of paging the instant a metric crosses its threshold, only fire when it stays above the threshold for a configured window (the for_seconds in the evaluator, equivalent to Prometheus’s for: clause). This suppresses one-off spikes so on-call engineers trust and act on the pages they do receive.

Why must dead-letter-queue labels avoid the full serial number? #

Because every distinct label value creates a separate time series. A serial (21) is high-cardinality by design — potentially millions of values — so labelling a counter by it can spawn millions of series and overwhelm the time-series database, turning your monitoring system into the outage. Label by partner GLN, GTIN root, event type, and a small set of enumerated reasons; keep serial-level detail in structured logs.

Is monitoring data itself subject to DSCSA retention? #

Parts of it are. Alert history and dead-letter drain records are audit artifacts that show a missing-record window was detected and remediated, and drain runs are chain-of-custody events that belong in the 21 CFR Part 11 audit trail. Compliance-facing metric rollups should be retained as evidence that verification SLAs were met across the six-year DSCSA window, even though raw high-resolution metrics are downsampled long before then.

Related #

• Building EPCIS Ingestion Dashboards with Python — query and layout patterns that make a stall obvious at a glance.
• Configuring SLA Alerts for Serialization Pipelines — turning service-level objectives into sustained-breach threshold rules.
• Draining Dead-Letter Queues Safely — bounded replay, poison-message handling, and drain-run audit records.
• Real-Time Event Stream Processing — the sibling area whose streams these metrics observe.
• Schema Validation & Error Handling — where the malformed events that fill the dead-letter queue originate.
• Serialization Data Ingestion & EPCIS Event Sync — the parent section this monitoring topic belongs to.