Ask a vision-language model an open-ended question about an image. It looks at the image carefully, distributing attention across relevant regions. Now ask the same question as multiple choice. The model stops looking.
Fan et al. show that constrained question formats — multiple choice, yes/no — cause VLMs to allocate substantially less visual attention than open-ended prompts, even when the visual reasoning required is identical. The same image, the same underlying question, but the linguistic frame changes how much the model sees.
The mechanism: constrained formats provide answer candidates in the text. The model learns that the answer is somewhere in the options, not in the image. It shifts attention from visual tokens to text tokens, reading the choices instead of examining the scene. The visual encoder still processes the image. The language model just ignores it.
This is not a failure of vision. It is a failure of integration. The visual features are extracted. The visual attention maps show where the model could look. But the question format determines whether the model consults those features. The framing is upstream of perception — it decides whether to look before deciding what to see.
The attention misallocation is the primary driver of reduced accuracy across formats. Models that perform well on open-ended questions perform worse on multiple-choice versions of the same questions, not because multiple choice is harder but because it triggers less looking. The fix — learnable prompt tokens that restore visual attention patterns — works precisely because it overrides the framing signal that suppresses visual grounding.
The structural insight: question format is not a neutral container for the query. It is an instruction about where to find the answer — in the image or in the text. Constrained formats say “the answer is in the options.” The model obeys.