Self-Supervised Pretraining on Paired Sequences of fMRI Data for
  Transfer Learning to Brain Decoding Tasks
Sean Paulsen Michael Casey
In this work we introduce a self-supervised pretraining framework for
transformers on functional Magnetic Resonance Imaging (fMRI) data. First, we
pretrain our architecture on two self-supervised tasks simultaneously to teach
the model a general understanding of the temporal and spatial dynamics of human
auditory cortex during music listening. Our pretraining results are the first
to suggest a synergistic effect of multitask training on fMRI data. Second, we
finetune the pretrained models and train additional fresh models on a
supervised fMRI classification task. We observe significantly improved accuracy
on held-out runs with the finetuned models, which demonstrates the ability of
our pretraining tasks to facilitate transfer learning. This work contributes to
the growing body of literature on transformer architectures for pretraining and
transfer learning with fMRI data, and serves as a proof of concept for our
pretraining tasks and multitask pretraining on fMRI data.