Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding
  (Survey)
Subba Reddy Oota Manish Gupta Raju S. Bapi Gael Jobard Frederic Alexandre Xavier Hinaut
How does the brain represent different modes of information? Can we design a
system that automatically understands what the user is thinking? Such questions
can be answered by studying brain recordings like functional magnetic resonance
imaging (fMRI). As a first step, the neuroscience community has contributed
several large cognitive neuroscience datasets related to passive
reading/listening/viewing of concept words, narratives, pictures and movies.
Encoding and decoding models using these datasets have also been proposed in
the past two decades. These models serve as additional tools for basic research
in cognitive science and neuroscience. Encoding models aim at generating fMRI
brain representations given a stimulus automatically. They have several
practical applications in evaluating and diagnosing neurological conditions and
thus also help design therapies for brain damage. Decoding models solve the
inverse problem of reconstructing the stimuli given the fMRI. They are useful
for designing brain-machine or brain-computer interfaces. Inspired by the
effectiveness of deep learning models for natural language processing, computer
vision, and speech, recently several neural encoding and decoding models have
been proposed. In this survey, we will first discuss popular representations of
language, vision and speech stimuli, and present a summary of neuroscience
datasets. Further, we will review popular deep learning based encoding and
decoding architectures and note their benefits and limitations. Finally, we
will conclude with a brief summary and discussion about future trends. Given
the large amount of recently published work in the `computational cognitive
neuroscience' community, we believe that this survey nicely organizes the
plethora of work and presents it as a coherent story.