MindDiffuser: Controlled Image Reconstruction from Human Brain Activity
  with Semantic and Structural Diffusion
Yizhuo Lu Changde Du Qiongyi zhou Dianpeng Wang Huiguang He
Reconstructing visual stimuli from brain recordings has been a meaningful and
challenging task. Especially, the achievement of precise and controllable image
reconstruction bears great significance in propelling the progress and
utilization of brain-computer interfaces. Despite the advancements in complex
image reconstruction techniques, the challenge persists in achieving a cohesive
alignment of both semantic (concepts and objects) and structure (position,
orientation, and size) with the image stimuli. To address the aforementioned
issue, we propose a two-stage image reconstruction model called MindDiffuser.
In Stage 1, the VQ-VAE latent representations and the CLIP text embeddings
decoded from fMRI are put into Stable Diffusion, which yields a preliminary
image that contains semantic information. In Stage 2, we utilize the CLIP
visual feature decoded from fMRI as supervisory information, and continually
adjust the two feature vectors decoded in Stage 1 through backpropagation to
align the structural information. The results of both qualitative and
quantitative analyses demonstrate that our model has surpassed the current
state-of-the-art models on Natural Scenes Dataset (NSD). The subsequent
experimental findings corroborate the neurobiological plausibility of the
model, as evidenced by the interpretability of the multimodal feature employed,
which align with the corresponding brain responses.