Monitor a deployed model long enough, and any fixed statistical test will eventually raise a false alarm — even if nothing has changed. The longer you watch, the more certain the false alarm becomes. This is the fundamental trap of model monitoring: the act of continuous observation violates the assumptions of the test being applied.
Farran addresses this with PITMonitor, which uses a mixture e-process to provide Type I error control over an unbounded monitoring horizon. The key insight is that probability integral transforms — checking whether a model's predicted probabilities are calibrated — can be combined with anytime-valid sequential testing to create a monitor that never makes a false claim regardless of how long it runs.
The distinction matters because current practice ignores it. Most deployed model monitoring uses tests designed for fixed sample sizes, applied repeatedly at arbitrary intervals. The resulting false alarm rate is not the nominal one — it accumulates, and eventually the monitor cries wolf on a model that hasn't moved. The response is either to ignore the monitor (defeating its purpose) or to recalibrate models that were fine (wasting effort and potentially degrading performance).
The trade-off is honest: detection delay under local drift is “substantially longer” than aggressive baselines. You pay for the guarantee of no false alarms with slower detection of real problems. But this is the correct trade-off for production systems where false alarms erode trust in the monitoring infrastructure itself. A monitor nobody believes is worse than no monitor at all.