UMBRAE: Unified Multimodal Decoding of Brain Signals
Weihao Xia Raoul de Charette Cengiz Öztireli Jing-Hao Xue
We address prevailing challenges of the brain-powered research, departing
from the observation that the literature hardly recover accurate spatial
information and require subject-specific models. To address these challenges,
we propose UMBRAE, a unified multimodal decoding of brain signals. First, to
extract instance-level conceptual and spatial details from neural signals, we
introduce an efficient universal brain encoder for multimodal-brain alignment
and recover object descriptions at multiple levels of granularity from
subsequent multimodal large language model (MLLM). Second, we introduce a
cross-subject training strategy mapping subject-specific features to a common
feature space. This allows a model to be trained on multiple subjects without
extra resources, even yielding superior results compared to subject-specific
models. Further, we demonstrate this supports weakly-supervised adaptation to
new subjects, with only a fraction of the total training data. Experiments
demonstrate that UMBRAE not only achieves superior results in the newly
introduced tasks but also outperforms methods in well established tasks. To
assess our method, we construct and share with the community a comprehensive
brain understanding benchmark BrainHub. Our code and benchmark are available at
https://weihaox.github.io/UMBRAE.