Language models learn grammatical distinctions from data. The prediction: more data, better grammar. The Bigram Hypothesis says something different: the order in which statistical patterns appear during training can lock in errors that no amount of subsequent data corrects.
Manna, Snyder, and Tabor (arXiv:2603.00359) trained Meta's OPT on the BabyLM dataset and evaluated it against 67 grammatical test classes from the BLiMP benchmark. In roughly one-third of cases, the model failed to consistently prefer grammatical over ungrammatical sentences — not because it hadn't seen enough data, but because it had seen the wrong statistics first. The errors weren't random. They were systematic and stable: the model established an incorrect grammatical distinction early in training and maintained it throughout.
The mechanism: bigram statistics — the frequency of adjacent word pairs — create an initial landscape of associations. Some of these associations happen to correlate with the wrong grammatical category. The model commits to this distinction before it has processed enough varied examples to discover the error. Once committed, the distinction becomes load-bearing: subsequent learning builds on it rather than correcting it. The early error is not overwritten because overwriting it would require unlearning the downstream structure that depends on it.
This is not a data volume problem. Feeding the model more training data of the same distribution deepens the entrenchment rather than eroding it. The error is structural, not statistical. It lives in the training trajectory, not in the training set.
The general pattern: in any system that learns incrementally and builds higher-order structure on top of lower-order distinctions, the timing of early exposure determines what can be learned later. An error in the foundation is harder to correct than an error in the superstructure — not because foundations are intrinsically rigid, but because everything built above them would collapse if the foundation shifted. The cost of correction exceeds the cost of the original mistake.
The implication for learning systems broadly: the critical window isn't when the most data arrives. It's when the first structural distinctions form. What you learn first constrains what you can learn at all.
Manna, Snyder, & Tabor, "How Large Language Models Get Stuck: Early structure with persistent errors," arXiv:2603.00359 (2026).