Brain Captioning: Decoding human brain activity into images and text
Matteo Ferrante Furkan Ozcelik Tommaso Boccato Rufin VanRullen Nicola Toschi
Every day, the human brain processes an immense volume of visual information,
relying on intricate neural mechanisms to perceive and interpret these stimuli.
Recent breakthroughs in functional magnetic resonance imaging (fMRI) have
enabled scientists to extract visual information from human brain activity
patterns. In this study, we present an innovative method for decoding brain
activity into meaningful images and captions, with a specific focus on brain
captioning due to its enhanced flexibility as compared to brain decoding into
images. Our approach takes advantage of cutting-edge image captioning models
and incorporates a unique image reconstruction pipeline that utilizes latent
diffusion models and depth estimation. We utilized the Natural Scenes Dataset,
a comprehensive fMRI dataset from eight subjects who viewed images from the
COCO dataset. We employed the Generative Image-to-text Transformer (GIT) as our
backbone for captioning and propose a new image reconstruction pipeline based
on latent diffusion models. The method involves training regularized linear
regression models between brain activity and extracted features. Additionally,
we incorporated depth maps from the ControlNet model to further guide the
reconstruction process. We evaluate our methods using quantitative metrics for
both generated captions and images. Our brain captioning approach outperforms
existing methods, while our image reconstruction pipeline generates plausible
images with improved spatial relationships. In conclusion, we demonstrate
significant progress in brain decoding, showcasing the enormous potential of
integrating vision and language to better understand human cognition. Our
approach provides a flexible platform for future research, with potential
applications in various fields, including neural art, style transfer, and
portable devices.