Visual-Attribute Prompt Learning for Progressive Mild Cognitive
  Impairment Prediction
Luoyao Kang Haifan Gong Xiang Wan Haofeng Li
Deep learning (DL) has been used in the automatic diagnosis of Mild Cognitive
Impairment (MCI) and Alzheimer's Disease (AD) with brain imaging data. However,
previous methods have not fully exploited the relation between brain image and
clinical information that is widely adopted by experts in practice. To exploit
the heterogeneous features from imaging and tabular data simultaneously, we
propose the Visual-Attribute Prompt Learning-based Transformer (VAP-Former), a
transformer-based network that efficiently extracts and fuses the multi-modal
features with prompt fine-tuning. Furthermore, we propose a Prompt fine-Tuning
(PT) scheme to transfer the knowledge from AD prediction task for progressive
MCI (pMCI) diagnosis. In details, we first pre-train the VAP-Former without
prompts on the AD diagnosis task and then fine-tune the model on the pMCI
detection task with PT, which only needs to optimize a small amount of
parameters while keeping the backbone frozen. Next, we propose a novel global
prompt token for the visual prompts to provide global guidance to the
multi-modal representations. Extensive experiments not only show the
superiority of our method compared with the state-of-the-art methods in pMCI
prediction but also demonstrate that the global prompt can make the prompt
learning process more effective and stable. Interestingly, the proposed prompt
learning model even outperforms the fully fine-tuning baseline on transferring
the knowledge from AD to pMCI.