Psychometry: An Omnifit Model for Image Reconstruction from Human Brain
  Activity
Ruijie Quan Wenguan Wang Zhibo Tian Fan Ma Yi Yang
Reconstructing the viewed images from human brain activity bridges human and
computer vision through the Brain-Computer Interface. The inherent variability
in brain function between individuals leads existing literature to focus on
acquiring separate models for each individual using their respective brain
signal data, ignoring commonalities between these data. In this article, we
devise Psychometry, an omnifit model for reconstructing images from functional
Magnetic Resonance Imaging (fMRI) obtained from different subjects. Psychometry
incorporates an omni mixture-of-experts (Omni MoE) module where all the experts
work together to capture the inter-subject commonalities, while each expert
associated with subject-specific parameters copes with the individual
differences. Moreover, Psychometry is equipped with a retrieval-enhanced
inference strategy, termed Ecphory, which aims to enhance the learned fMRI
representation via retrieving from prestored subject-specific memories. These
designs collectively render Psychometry omnifit and efficient, enabling it to
capture both inter-subject commonality and individual specificity across
subjects. As a result, the enhanced fMRI representations serve as conditional
signals to guide a generation model to reconstruct high-quality and realistic
images, establishing Psychometry as state-of-the-art in terms of both
high-level and low-level metrics.