Diffusion models learn to generate images by reversing a noise process. But what do they learn first? Biroli et al. (arXiv:2603.12901) show that standard diffusion models exhibit a distributional simplicity bias: they learn pairwise statistics (correlations between pixels) before higher-order statistics (textures, edges, complex structures).
The mechanism is sample complexity. Pairwise statistics require linear sample complexity — O(d) samples for d-dimensional data. Fourth-order statistics require at least cubic complexity — O(d³) samples. The model doesn't choose to learn simple things first; the gradient descent dynamics naturally converge on the statistics that can be estimated from fewer samples.
The authors introduce the “diffusion information exponent,” a mathematical invariant that predicts how many training samples a given statistical feature needs before the model captures it. This exponent determines the learning order: features with lower exponents are learned first, regardless of their perceptual importance.
There's an exception. When higher-order statistics share structure with lower-order ones — when the fourth-order correlations can be decomposed in terms of the second-order ones — the sample complexity drops back to linear. Structure in the statistics shortcuts the learning hierarchy.
The parallel to human perception is suggestive. Human visual development also proceeds from simple to complex: infants detect contrast and edges before textures and objects. The analogy is imperfect — the mechanisms differ entirely — but the ordering is the same. Simple statistics are learned first because they're learnable first, independent of whether the system is a neural network or a visual cortex.
The practical implication: if your training set is small, your diffusion model has learned the pairwise statistics and little else. The generated images will have correct color distributions and spatial correlations but wrong textures and structures. The model isn't failing randomly — it's failing in a predictable order determined by the information exponent.