When a language model says “I'm not sure, but I think the answer is X,” where does the uncertainty come from? The naive model: the model generates X, evaluates its confidence in X, and wraps the output in hedging language proportional to its uncertainty. Confidence is computed on-demand, in real time, as a post-hoc assessment of the generated content.
The mechanism is different (arXiv:2603.17839). Confidence is cached during generation, not computed after it. The model's internal representations encode a confidence signal early in the generation process — before the hedging language appears, sometimes before the factual content itself. The verbal expression of confidence (“I think,” “I'm fairly certain,” “approximately”) is downstream of an internal state that was already formed, not a fresh evaluation of the output.
This means the model's self-reported confidence is not cheap talk. It reflects a genuine internal signal — one that correlates with the model's actual probability of being correct. Models that say “I'm confident” are, on average, more often right than models that say “I'm unsure.” The calibration is imperfect, but the signal is real.
The implication is architecturally specific: confidence is not a separate computation layered on top of generation. It is integrated into the same forward pass that produces the answer. The model does not first answer and then decide how confident to be. It answers and calibrates simultaneously, with both processes sharing the same intermediate representations.
This makes confidence reports both more trustworthy and less transparent than they appear. More trustworthy: the signal is genuine, not confabulated. Less transparent: the confidence is not the result of deliberate introspection but of a cached state that the model cannot articulate the origins of. The model knows how confident it is before it knows why.