Show a multimodal LLM a chemical structure diagram. Ask it to identify the molecule. It fails. Now ask it to reason about the molecule's properties given its name. It succeeds.
Li et al. evaluate leading multimodal models across five symbolic domains — language, culture, mathematics, physics, chemistry — and find a systematic cognitive mismatch: models fail at basic symbol recognition but succeed at complex reasoning tasks that depend on those same symbols.
The explanation: the models are not seeing the symbols at all. They are pattern-matching the visual input to a textual approximation, then reasoning over the text. When the visual-to-text mapping works (the symbol looks like something in training data), recognition succeeds. When it doesn't (unusual notation, handwritten formulas, culturally specific characters), recognition fails. But the reasoning that follows — given the correct name or description — remains strong because it operates entirely in language space.
The mismatch reveals a dissociation between perception and cognition in multimodal AI. The vision encoder extracts features. The language model reasons over features. But the features extracted from discrete symbols — mathematical formulas, chemical diagrams, musical notation — are unreliable. The models' linguistic probability patterns fill in what visual perception fails to provide, producing correct reasoning from incorrect perception.
The structural implication: multimodal LLMs have not learned to see symbols. They have learned to guess what symbols say based on visual context, then reason as if the guess were correct. When the guess is right, the system appears competent. When it is wrong, the error is invisible — the reasoning proceeds fluently from a misread input, producing confident wrong answers.