Shannon entropy is usually introduced as a measure of uncertainty. A fair coin has 1 bit of entropy because you cannot predict the outcome. A loaded coin has less. A certain outcome has zero. The standard interpretation frames entropy in epistemic terms: it quantifies what you do not know.
Bradley and Vigneaux (arXiv:2501.06662) offer a different derivation that reframes what entropy is measuring. They take the probability distributions produced by a language model — the next-token probabilities for each prompt — and use them to define a metric space enriched over the unit interval. Each text (prompt) is a point. The distance between two points is determined by how differently the model continues them. They then compute the magnitude of this space, a categorical invariant that measures the “effective number of points.”
The magnitude function turns out to equal the sum of Tsallis entropies of the next-token distributions, plus the cardinality of the output vocabulary. Take the derivative at the right parameter value and Shannon entropy falls out. Entropy is not an additional measurement imposed on the space. It is the magnitude — the effective size — of the space that the model's probability distributions define.
The reframing is concrete. A language model with high entropy on a given prompt is not “uncertain” in any epistemically interesting sense. It is operating in a region of text-space that is effectively large — many continuations are equally viable, so the local neighborhood has high effective dimension. Low entropy means the space has collapsed: few continuations remain, the neighborhood is small, the effective size is near one. What the model “knows” or “doesn't know” is irrelevant. The geometry of the probability landscape determines the number, and the number is the entropy.
The categorical machinery is not decorative. Magnitude is defined for enriched categories in general, and its specialization to metric spaces has been studied for two decades. What Bradley and Vigneaux show is that the same invariant, applied to the specific metric structure that language model probabilities create, recovers the information-theoretic quantities without assuming them. Entropy is not a property of the observer's ignorance. It is a property of the space's size as measured by the only ruler the space provides.