MindBridge: A Cross-Subject Brain Decoding Framework
Shizun Wang Songhua Liu Zhenxiong Tan Xinchao Wang
Brain decoding, a pivotal field in neuroscience, aims to reconstruct stimuli
from acquired brain signals, primarily utilizing functional magnetic resonance
imaging (fMRI). Currently, brain decoding is confined to a
per-subject-per-model paradigm, limiting its applicability to the same
individual for whom the decoding model is trained. This constraint stems from
three key challenges: 1) the inherent variability in input dimensions across
subjects due to differences in brain size; 2) the unique intrinsic neural
patterns, influencing how different individuals perceive and process sensory
information; 3) limited data availability for new subjects in real-world
scenarios hampers the performance of decoding models. In this paper, we present
a novel approach, MindBridge, that achieves cross-subject brain decoding by
employing only one model. Our proposed framework establishes a generic paradigm
capable of addressing these challenges by introducing biological-inspired
aggregation function and novel cyclic fMRI reconstruction mechanism for
subject-invariant representation learning. Notably, by cycle reconstruction of
fMRI, MindBridge can enable novel fMRI synthesis, which also can serve as
pseudo data augmentation. Within the framework, we also devise a novel
reset-tuning method for adapting a pretrained model to a new subject.
Experimental results demonstrate MindBridge's ability to reconstruct images for
multiple subjects, which is competitive with dedicated subject-specific models.
Furthermore, with limited data for a new subject, we achieve a high level of
decoding accuracy, surpassing that of subject-specific models. This advancement
in cross-subject brain decoding suggests promising directions for wider
applications in neuroscience and indicates potential for more efficient
utilization of limited fMRI data in real-world scenarios. Project page:
https://littlepure2333.github.io/MindBridge