GAME: Generalized deep learning model towards multimodal data
  integration for early screening of adolescent mental disorders
Zhicheng Du Chenyao Jiang Xi Yuan Shiyao Zhai Zhengyang Lei Shuyue Ma Yang Liu Qihui Ye Chufan Xiao Qiming Huang Ming Xu Dongmei Yu Peiwu Qin
The timely identification of mental disorders in adolescents is a global
public health challenge.Single factor is difficult to detect the abnormality
due to its complex and subtle nature. Additionally, the generalized multimodal
Computer-Aided Screening (CAS) systems with interactive robots for adolescent
mental disorders are not available. Here, we design an android application with
mini-games and chat recording deployed in a portable robot to screen 3,783
middle school students and construct the multimodal screening dataset,
including facial images, physiological signs, voice recordings, and textual
transcripts.We develop a model called GAME (Generalized Model with Attention
and Multimodal EmbraceNet) with novel attention mechanism that integrates
cross-modal features into the model. GAME evaluates adolescent mental
conditions with high accuracy (73.34%-92.77%) and F1-Score (71.32%-91.06%).We
find each modality contributes dynamically to the mental disorders screening
and comorbidities among various mental disorders, indicating the feasibility of
explainable model. This study provides a system capable of acquiring multimodal
information and constructs a generalized multimodal integration algorithm with
novel attention mechanisms for the early screening of adolescent mental
disorders.