Language models trained on next-token prediction appear to prefer true statements over false ones. Given a choice between “2 + 2 = 4” and “2 + 2 = 5,” the model assigns higher probability to the correct answer. This looks like knowledge. It might not be.
Krestnikov (arXiv:2603.11749, March 2026) argues that the mechanism is compression, not truth-seeking. Next-token prediction minimizes description length — the model learns to represent its training data as compactly as possible. True statements compress better than false ones when the errors in the training data are random. A rule like “2 + 2 = 4” allows the model to compress all consistent examples into a single pattern. Random errors — “2 + 2 = 5” here, “3 + 3 = 8” there — require separate encoding for each deviation. Truth wins because it is regular. Falsehood loses because it is scattered.
The experimental test: replace random errors with a coherent but incorrect rule system. Make the falsehoods internally consistent — an alternative arithmetic where the mistakes follow their own logic. Now the false system compresses just as well as the true one. The model's preference for correctness drops to near chance. It cannot distinguish organized falsehood from truth, because both are equally compressible.
The implication is that the model's apparent accuracy is a side effect of the structure of real-world errors, not of an alignment between compression and reality. Errors in real training data are predominantly uncoordinated — typos, mistakes, misattributions — and uncoordinated errors are harder to compress than the truth they corrupt. The model exploits this asymmetry. But the asymmetry is contingent. If someone deliberately introduced systematic falsehoods — a consistent alternative history, a coherent pseudoscience, a well-organized conspiracy theory — the compression objective would not resist. The model compresses patterns. It does not check whether the patterns correspond to anything.
This explains a specific failure mode: language models are more vulnerable to coordinated misinformation than to random noise. Random noise is statistically obvious — it degrades compression. Coordinated misinformation is statistically invisible — it compresses as well as truth. The defense that training data is “mostly correct” protects against scattered errors but not against internally consistent fictions.
Krestnikov, "Compression Favors Consistency, Not Truth: When and Why Language Models Prefer Correct Information," arXiv:2603.11749 (March 2026).