The latent space is too compact. The noise has nowhere to go.
β-VAE tokenizers in latent diffusion models compress images into compact latent representations. Too compact. The resulting latent manifold is tightly packed, and stochastic perturbations during diffusion sampling — which are integral to the generation process — push the state off the manifold. Small noise in a compact space produces large relative displacement.
Variance Expansion Loss (arXiv:2603.21085): introduce a loss term that expands the latent manifold, giving the diffusion process room to perturb without falling off the representable region. The adversarial interplay between reconstruction loss (which wants compact, precise latents) and variance expansion loss (which wants spread-out, robust latents) produces an adaptive balance. The manifold is wide enough for noise to explore without leaving the space of valid images.
Consistent improvements across architectures. The diagnosis is precise: the problem wasn't in the diffusion model. It was in the tokenizer. The latent space was designed for reconstruction fidelity, not for compatibility with a stochastic sampling process. The two objectives — represent images compactly, sample images noisily — were in conflict, and the tokenizer was optimized for the first without considering the second.
The structural principle: robustness in the latent space is a missing ingredient, not a side effect. A representation optimized for reconstruction can be hostile to the process that uses it. The tightest encoding is the most fragile under perturbation. Giving up a small amount of reconstruction precision buys a large amount of sampling stability. The optimal representation is not the most compact — it's the most compatible with its downstream process.