Brainformer: Modeling MRI Brain Functions to Machine Vision
Xuan-Bac Nguyen Xin Li Samee U. Khan Khoa Luu
"Perception is reality". Human perception plays a vital role in forming
beliefs and understanding reality. Exploring how the human brain works in the
visual system facilitates bridging the gap between human visual perception and
computer vision models. However, neuroscientists study the brain via
Neuroimaging, i.e., Functional Magnetic Resonance Imaging (fMRI), to discover
the brain's functions. These approaches face interpretation challenges where
fMRI data can be complex and require expertise. Therefore, neuroscientists make
inferences about cognitive processes based on patterns of brain activities,
which can lead to potential misinterpretation or limited functional
understanding. In this work, we first present a simple yet effective
Brainformer approach, a novel Transformer-based framework, to analyze the
patterns of fMRI in the human perception system from the machine learning
perspective. Secondly, we introduce a novel mechanism incorporating fMRI, which
represents the human brain activities, as the supervision for the machine
vision model. This work also introduces a novel perspective on transferring
knowledge from human perception to neural networks. Through our experiments, we
demonstrated that by leveraging fMRI information, the machine vision model can
achieve potential results compared to the current State-of-the-art methods in
various image recognition tasks.