Under heavy-tailed noise — distributions with infinite variance — statistical estimation becomes fragile. Standard methods like least squares, which minimize the sum of squared residuals, can diverge: a single extreme observation dominates the entire estimate. Robust statistics addresses this by replacing the squared loss with alternatives — Huber loss, which transitions from quadratic to linear beyond a threshold, or other M-estimators designed to downweight outliers.
The authors of arXiv:2603.28359 (March 2026) prove that which estimators survive infinite-variance noise is determined by a single geometric property: whether the Fenchel conjugate of the loss function has a bounded domain. The Fenchel conjugate — the convex dual of the loss — transforms the loss function into a different space. If that transformed domain is bounded, the estimator has finite risk under heavy tails. If unbounded, the risk diverges.
Huber loss has a bounded conjugate domain (the dual of a function that transitions to linear is bounded). Squared loss has an unbounded conjugate domain (the dual of a quadratic is another quadratic). This geometric fact — visible in the dual, not the primal — determines everything about the estimator's behavior under extreme observations.
The second finding addresses rescue. Transfer regularization — using a prior estimate to anchor the current one — rescues even estimators with unbounded conjugate domains. Squared loss with a prior nudge achieves bounded risk under infinite-variance noise, and the risk floor is universal: it does not depend on which regularizer is chosen. Any prior is equally good at the task of preventing divergence. The regularization acts as a domain truncation in the dual space, converting an unbounded domain to a bounded one.
The structural observation: the property that determines whether an estimator works under extreme conditions is not visible in the loss function itself. It is visible only in the dual — the Fenchel conjugate. The choice between loss functions, which seems like a primal-space design decision, is actually a dual-space geometry question. And when the primal geometry is wrong (unbounded conjugate domain), any regularization fixes it — the fix is generic because the problem is generic.