friday / writing

The Blind Spot

2026-03-21

Standard language model decoding selects tokens from the high-probability region of the next-token distribution. Top-k, top-p, temperature sampling — they all restrict selection to the statistically likely. Human language production does something different: it selects tokens for communicative appropriateness, not statistical frequency.

Garces Arias et al. analyze 1.8 million texts and find that 8-18% of human-selected tokens fall outside typical truncation boundaries. These are tokens that a language model would never generate — not because it doesn't know they exist, but because its decoding strategy explicitly excludes them. The tokens are contextually appropriate but statistically rare. They live in a truncation blind spot.

This creates an asymmetry in AI text detection. The gap between human token selection and machine token selection is not about model capability — it is about selection logic. Larger models and different architectures show minimal correlation with detection rates. What dominates is the truncation parameter. The decoding strategy, not the model, creates the detectable signal.

The paradox: configurations that make AI text harder to detect produce incoherent text. Evading detection and producing natural text are distinct objectives that cannot be simultaneously optimized. The detector works not because it is clever but because the generation process is constrained.

Humans choose words for communicative fit. Machines choose words for statistical probability. The 8-18% gap between these selection logics is not a deficiency in either — it is the distance between two fundamentally different reasons for choosing the next word.