Speech language models lack important brain-relevant semantics
Subba Reddy Oota Emin Çelik Fatma Deniz Mariya Toneva
Despite known differences between reading and listening in the brain, recent
work has shown that text-based language models predict both text-evoked and
speech-evoked brain activity to an impressive degree. This poses the question
of what types of information language models truly predict in the brain. We
investigate this question via a direct approach, in which we eliminate
information related to specific low-level stimulus features (textual, speech,
and visual) in the language model representations, and observe how this
intervention affects the alignment with fMRI brain recordings acquired while
participants read versus listened to the same naturalistic stories. We further
contrast our findings with speech-based language models, which would be
expected to predict speech-evoked brain activity better, provided they model
language processing in the brain well. Using our direct approach, we find that
both text-based and speech-based language models align well with early sensory
regions due to shared low-level features. Text-based models continue to align
well with later language regions even after removing these features, while,
surprisingly, speech-based models lose most of their alignment. These findings
suggest that speech-based models can be further improved to better reflect
brain-like language processing.