JGAT: a joint spatio-temporal graph attention model for brain decoding
Han Yi Chiu Liang Zhao Anqi Wu
The decoding of brain neural networks has been an intriguing topic in
neuroscience for a well-rounded understanding of different types of brain
disorders and cognitive stimuli. Integrating different types of connectivity,
e.g., Functional Connectivity (FC) and Structural Connectivity (SC), from
multi-modal imaging techniques can take their complementary information into
account and therefore have the potential to get better decoding capability.
However, traditional approaches for integrating FC and SC overlook the
dynamical variations, which stand a great chance to over-generalize the brain
neural network. In this paper, we propose a Joint kernel Graph Attention
Network (JGAT), which is a new multi-modal temporal graph attention network
framework. It integrates the data from functional Magnetic Resonance Images
(fMRI) and Diffusion Weighted Imaging (DWI) while preserving the dynamic
information at the same time. We conduct brain-decoding tasks with our JGAT on
four independent datasets: three of 7T fMRI datasets from the Human Connectome
Project (HCP) and one from animal neural recordings. Furthermore, with
Attention Scores (AS) and Frame Scores (FS) computed and learned from the
model, we can locate several informative temporal segments and build meaningful
dynamical pathways along the temporal domain for the HCP datasets. The URL to
the code of JGAT model: https://github.com/BRAINML-GT/JGAT.