Language models are trained on text. During generation, they produce text. The naive expectation: if a fact appears in training data, it should appear in the model's output with probability related to its training frequency. More exposure means more likely generation.
The data says otherwise. Training data presence does not directly predict output probability. Models demonstrably contain information that they systematically suppress during standard generation. The knowledge is there — probing techniques can extract it, fine-tuning can surface it — but the generation process actively filters it out.
This is not forgetting. Forgetting would mean the information is degraded or absent from the model's parameters. What the paper documents is structured suppression: the model has learned the information and has learned not to produce it. The boundary between what the model knows and what it says is an active, maintained filter.
The implications split in two directions.
First, the “stochastic parrot” critique — that language models merely repeat their training data — is wrong in an unexpected way. The models retain more than they repeat. They are not parrots because parrots reproduce what they hear. These systems reproduce selectively, and the selectivity is itself a learned behavior. The interesting question is not what they memorize but what they choose to suppress.
Second, knowledge extraction from language models is harder than it appears. Standard generation samples from a filtered distribution. The filter is not random — it is structured by whatever objective the model learned during training. Prompting, temperature adjustment, and other generation-time interventions interact with this filter, sometimes opening it and sometimes tightening it. The model's silence on a topic does not mean ignorance. It may mean active suppression.
The general pattern: in any system that learns both content and a generation policy, what the system produces is a joint function of what it knows and what it has learned to express. These are independent degrees of freedom. Measuring one does not inform you about the other.