MulCogBench: A Multi-modal Cognitive Benchmark Dataset for Evaluating
  Chinese and English Computational Language Models
Yunhao Zhang Xiaohan Zhang Chong Li Shaonan Wang Chengqing Zong
Pre-trained computational language models have recently made remarkable
progress in harnessing the language abilities which were considered unique to
humans. Their success has raised interest in whether these models represent and
process language like humans. To answer this question, this paper proposes
MulCogBench, a multi-modal cognitive benchmark dataset collected from native
Chinese and English participants. It encompasses a variety of cognitive data,
including subjective semantic ratings, eye-tracking, functional magnetic
resonance imaging (fMRI), and magnetoencephalography (MEG). To assess the
relationship between language models and cognitive data, we conducted a
similarity-encoding analysis which decodes cognitive data based on its pattern
similarity with textual embeddings. Results show that language models share
significant similarities with human cognitive data and the similarity patterns
are modulated by the data modality and stimuli complexity. Specifically,
context-aware models outperform context-independent models as language stimulus
complexity increases. The shallow layers of context-aware models are better
aligned with the high-temporal-resolution MEG signals whereas the deeper layers
show more similarity with the high-spatial-resolution fMRI. These results
indicate that language models have a delicate relationship with brain language
representations. Moreover, the results between Chinese and English are highly
consistent, suggesting the generalizability of these findings across languages.