Larger language models have larger context windows. More context should mean better architectural understanding — the model can see more of the codebase at once, identify more dependencies, maintain a richer mental map of the system's structure.
Sapunov tested this by probing models' beliefs about codebase architecture across multiple interactions. The finding: larger models suffer worse belief deterioration. They lose previously identified components between probes. Smaller models, with their limited context, maintain perfect consistency in their architectural maps.
The mechanism is interference. A larger context window contains more information, but it also contains more noise, more irrelevant detail, more competing signals. When the model re-examines the codebase after an intervening query, the additional context provides new evidence that can contradict the old map. The larger model integrates this new evidence — sometimes correctly, sometimes not — and the integration can overwrite previously correct beliefs.
The smaller model cannot hold the contradicting evidence. Its limited window forces it to rely on whatever summary it formed initially, and that summary, once formed, is stable precisely because it cannot be updated by new context. The limitation is the mechanism of consistency.
This is not a training data problem or a capability deficiency. It is a structural consequence of how attention mechanisms process long context. More context means more pathways for interference between early and late information. The model's belief at token N depends on all tokens before it, and in a long context, the dependencies are dense enough that perturbations propagate unpredictably.
Active exploration — the model navigating the codebase rather than receiving it passively — helps some models but harms others. The exploration generates new tokens that become part of the context, and these tokens can either reinforce or corrupt the existing architectural map depending on the model's sensitivity to self-generated interference.
The assumption that scale solves comprehension is wrong for sustained architectural reasoning. Bigger models see more but remember less consistently. The context window that enables broad vision also enables broad forgetting.