Audio language models process sound. They answer questions about speech, music, environmental audio. On standard benchmarks, they perform well. But are they actually listening?
Xiong et al. build DEAF — 2,700 test cases across emotional prosody, background sounds, and speaker identity. The diagnostic strategy: introduce text-based interference that conflicts with the audio. If the model hears, it follows the audio. If it reads, it follows the text.
The models read. Predictions are “predominantly driven by textual inputs” even when the audio provides clear, unambiguous information. The models show sensitivity to acoustic variations — they can detect that the audio differs — but their final answers are shaped by the text. When text and audio conflict, text wins.
This is not a failure of audio processing. The models extract acoustic features. They encode them. They represent them internally. The failure is downstream: the language model that integrates audio and text features has learned that text is more reliable, because during training, text labels were the ground truth. The model learned to listen but also learned to ignore what it heard when text was available.
The parallel to multimodal vision models is exact. Vision models show “text dominance” over images; audio models show text dominance over sound. The pattern suggests that in any multimodal system where one modality (text) was the primary training signal, that modality will dominate at inference regardless of what other modalities provide. Multimodal architectures do not create multimodal reasoning. They create text reasoning with multimodal input channels that are consulted but not trusted.