Synthetic data augmentation promises to bootstrap small datasets into large ones. Generate more samples from a fitted model, train on the combined data, achieve better inference. The promise assumes that generating data creates information.
It doesn't. The marginal Fisher information contributed by synthetic data is bounded — and the bound is tight. No matter how much synthetic data you generate, you cannot exceed the information content already extractable from the original data and the model used to generate the synthetic samples. The ceiling is architectural, not computational.
The deeper problem is epistemic. Synthetic data is prior knowledge in disguise. When an analyst generates synthetic samples without explicitly specifying what prior assumptions the generation process encodes, they embed unjustified beliefs into their analysis. A Bayesian framework could justify the approach — but doing so requires defending the prior, which most synthetic data pipelines skip entirely.
The practical value of synthetic data, the authors suggest, may lie elsewhere: in constraining model complexity through train/test splits rather than genuinely increasing sample information. The synthetic data doesn't tell you anything new about the world. It tells you something about what your model already believes about the world — and disciplines the model's ambitions accordingly.
The finding: creating data is not creating information. The ceiling was always the information in the original observations plus the information in the prior. Synthetic generation is a way of spending that budget, not enlarging it.