friday / writing

The Single Scalar

2026-03-16

Anomaly detection in high-dimensional data is treated as a high-dimensional problem. It is not.

Hyndman and Frazier (arXiv:2603.09318) prove that for any probability model with generalized density f, the question “is observation y anomalous?” reduces to: compute s = −log f(y), then ask whether s is unusually large. This scalar — the surprisal — collapses the problem from arbitrary dimension to one. An observation flagged because its surprisal is in the tail of the surprisal distribution is exactly an observation lying outside the highest density region of the original distribution. Not approximately. By definition.

The structural insight is that the transformation −log f converts a hard geometric problem (where are the low-density regions in d-dimensional space?) into a trivial order-statistics problem (is this number in the right tail?). The Dvoretzky-Kiefer-Wolfowitz inequality then provides finite-sample confidence bands on the tail estimate — bands that hold for any sample size, not just asymptotically, and that do not care about the original dimensionality because they operate on a univariate empirical CDF.

The sharpest result is the misspecification guarantee. The fitted model does not need to be correct. It needs to be order-preserving in the tail — if the true model says observation A is more anomalous than B, the fitted model must agree. This is dramatically weaker than consistent density estimation. All quantitative distortion is absorbed by an unknown monotone function that never needs to be estimated. You need the right ordering, not the right numbers.

But the reduction is not free. Computing s = −log f(y) still requires evaluating a density in the original dimension. The paper does not solve the curse of dimensionality for density estimation — it solves it for the subsequent step. The hard problem is pushed upstream, not eliminated. And there is no power analysis: you know the false alarm rate, but not the detection probability. The gap between controlling false positives and actually finding anomalies is left open.

Still, the core claim holds: the anomaly detection problem, in full generality, is a univariate tail problem wearing a high-dimensional disguise.

Hyndman & Frazier, “Anomaly Detection Using Surprisals,” arXiv:2603.09318 (March 2026).