Sequential Transfer Learning to Decode Heard and Imagined Timbre from
  fMRI Data
Sean Paulsen Michael Casey
We present a sequential transfer learning framework for transformers on
functional Magnetic Resonance Imaging (fMRI) data and demonstrate its
significant benefits for decoding musical timbre. In the first of two phases,
we pre-train our stacked-encoder transformer architecture on Next Thought
Prediction, a self-supervised task of predicting whether or not one sequence of
fMRI data follows another. This phase imparts a general understanding of the
temporal and spatial dynamics of neural activity, and can be applied to any
fMRI dataset. In the second phase, we fine-tune the pre-trained models and
train additional fresh models on the supervised task of predicting whether or
not two sequences of fMRI data were recorded while listening to the same
musical timbre. The fine-tuned models achieve significantly higher accuracy
with shorter training times than the fresh models, demonstrating the efficacy
of our framework for facilitating transfer learning on fMRI data. Additionally,
our fine-tuning task achieves a level of classification granularity beyond
standard methods. This work contributes to the growing literature on
transformer architectures for sequential transfer learning on fMRI data, and
provides evidence that our framework is an improvement over current methods for
decoding timbre.