Are words equally surprising in audio and audio-visual comprehension?
Pranava Madhyastha Ye Zhang Gabriella Vigliocco
We report a controlled study investigating the effect of visual information
(i.e., seeing the speaker) on spoken language comprehension. We compare the ERP
signature (N400) associated with each word in audio-only and audio-visual
presentations of the same verbal stimuli. We assess the extent to which
surprisal measures (which quantify the predictability of words in their lexical
context) are generated on the basis of different types of language models
(specifically n-gram and Transformer models) that predict N400 responses for
each word. Our results indicate that cognitive effort differs significantly
between multimodal and unimodal settings. In addition, our findings suggest
that while Transformer-based models, which have access to a larger lexical
context, provide a better fit in the audio-only setting, 2-gram language models
are more effective in the multimodal setting. This highlights the significant
impact of local lexical context on cognitive processing in a multimodal
environment.