NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual
  Pretraining and Multi-level Modulation
Jingyang Huo Yikai Wang Xuelin Qian Yun Wang Chong Li Jianfeng Feng Yanwei Fu
Recent fMRI-to-image approaches mainly focused on associating fMRI signals
with specific conditions of pre-trained diffusion models. These approaches,
while producing high-quality images, capture only a limited aspect of the
complex information in fMRI signals and offer little detailed control over
image creation. In contrast, this paper proposes to directly modulate the
generation process of diffusion models using fMRI signals. Our approach,
NeuroPictor, divides the fMRI-to-image process into three steps: i) fMRI
calibrated-encoding, to tackle multi-individual pre-training for a shared
latent space to minimize individual difference and enable the subsequent
cross-subject training; ii) fMRI-to-image cross-subject pre-training,
perceptually learning to guide diffusion model with high- and low-level
conditions across different individuals; iii) fMRI-to-image single-subject
refining, similar with step ii but focus on adapting to particular individual.
NeuroPictor extracts high-level semantic features from fMRI signals that
characterizing the visual stimulus and incrementally fine-tunes the diffusion
model with a low-level manipulation network to provide precise structural
instructions. By training with over 60,000 fMRI-image pairs from various
individuals, our model enjoys superior fMRI-to-image decoding capacity,
particularly in the within-subject setting, as evidenced in benchmark datasets.
Project page: https://jingyanghuo.github.io/neuropictor/.