Multimodal hierarchical multi-task deep learning framework for jointly
  predicting and explaining Alzheimer disease progression
Sayantan Kumar Sean Yu Thomas Kannampallil Andrew Michelson Aristeidis Sotiras Philip Payne
Early identification of Mild Cognitive Impairment (MCI) subjects who will
eventually progress to Alzheimer Disease (AD) is challenging. Existing deep
learning models are mostly single-modality single-task models predicting risk
of disease progression at a fixed timepoint. We proposed a multimodal
hierarchical multi-task learning approach which can monitor the risk of disease
progression at each timepoint of the visit trajectory. Longitudinal visit data
from multiple modalities (MRI, cognition, and clinical data) were collected
from MCI individuals of the Alzheimer Disease Neuroimaging Initiative (ADNI)
dataset. Our hierarchical model predicted at every timepoint a set of
neuropsychological composite cognitive function scores as auxiliary tasks and
used the forecasted scores at every timepoint to predict the future risk of
disease. Relevance weights for each composite function provided explanations
about potential factors for disease progression. Our proposed model performed
better than state-of-the-art baselines in predicting AD progression risk and
the composite scores. Ablation study on the number of modalities demonstrated
that imaging and cognition data contributed most towards the outcome. Model
explanations at each timepoint can inform clinicians 6 months in advance the
potential cognitive function decline that can lead to progression to AD in
future. Our model monitored their risk of AD progression every 6 months
throughout the visit trajectory of individuals. The hierarchical learning of
auxiliary tasks allowed better optimization and allowed longitudinal
explanations for the outcome. Our framework is flexible with the number of
input modalities and the selection of auxiliary tasks and hence can be
generalized to other clinical problems too.