Bayesian model checking uses leave-one-out predictive distributions: for each observation, predict it using a model fitted to all other observations. If the model is correct, the cumulative predictive probability of each left-out observation — the LOO-PIT value — should be uniformly distributed. Deviation from uniformity signals misfit.
Bürkner, Gabry, and Vehtari prove this check contains a hidden error. The LOO-PIT values are not independent. Each one is computed from nearly the same dataset (all but one observation), so the predictive distributions are highly correlated. The dependencies are substantial in finite samples and correlate with model complexity: more complex models produce more correlated LOO-PIT values because their predictions are more sensitive to individual data points.
Standard uniformity tests assume independence. When applied to dependent LOO-PIT values, they under-reject bad models and over-reject acceptable ones — the test's power is miscalibrated in both directions. The authors develop three testing procedures designed for dependent uniform values and an automated visualization for localized departures from uniformity. Their tests outperform standard tests precisely because they account for the correlation structure rather than treating each LOO-PIT value as an independent draw.
The through-claim is about the failure mode of recycled data. Leave-one-out resampling produces N predictive checks from N nearly identical datasets. The checks look independent — each uses a different held-out point — but share N−1 observations. The shared data creates correlations invisible to tests that examine marginal distributions only. The individual checks are valid. Their joint distribution is not what independent samples would produce. Testing one at a time works. Testing all at once requires knowing they are coupled.