Improving visual image reconstruction from human brain activity using
  latent diffusion models via multiple decoded inputs
Yu Takagi Shinji Nishimoto
The integration of deep learning and neuroscience has been advancing rapidly,
which has led to improvements in the analysis of brain activity and the
understanding of deep learning models from a neuroscientific perspective. The
reconstruction of visual experience from human brain activity is an area that
has particularly benefited: the use of deep learning models trained on large
amounts of natural images has greatly improved its quality, and approaches that
combine the diverse information contained in visual experiences have
proliferated rapidly in recent years. In this technical paper, by taking
advantage of the simple and generic framework that we proposed (Takagi and
Nishimoto, CVPR 2023), we examine the extent to which various additional
decoding techniques affect the performance of visual experience reconstruction.
Specifically, we combined our earlier work with the following three techniques:
using decoded text from brain activity, nonlinear optimization for structural
image reconstruction, and using decoded depth information from brain activity.
We confirmed that these techniques contributed to improving accuracy over the
baseline. We also discuss what researchers should consider when performing
visual reconstruction using deep generative models trained on large datasets.
Please check our webpage at
https://sites.google.com/view/stablediffusion-with-brain/. Code is also
available at https://github.com/yu-takagi/StableDiffusionReconstruction.