UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive
  signals and human language
Nuwa Xi Sendong Zhao Haochun Wang Chi Liu Bing Qin Ting Liu
Decoding text stimuli from cognitive signals (e.g. fMRI) enhances our
understanding of the human language system, paving the way for building
versatile Brain-Computer Interface. However, existing studies largely focus on
decoding individual word-level fMRI volumes from a restricted vocabulary, which
is far too idealized for real-world application. In this paper, we propose
fMRI2text, the first openvocabulary task aiming to bridge fMRI time series and
human language. Furthermore, to explore the potential of this new task, we
present a baseline solution, UniCoRN: the Unified Cognitive Signal
ReconstructioN for Brain Decoding. By reconstructing both individual time
points and time series, UniCoRN establishes a robust encoder for cognitive
signals (fMRI & EEG). Leveraging a pre-trained language model as decoder,
UniCoRN proves its efficacy in decoding coherent text from fMRI series across
various split settings. Our model achieves a 34.77% BLEU score on fMRI2text,
and a 37.04% BLEU when generalized to EEGto-text decoding, thereby surpassing
the former baseline. Experimental results indicate the feasibility of decoding
consecutive fMRI volumes, and the effectiveness of decoding different cognitive
signals using a unified structure.