An Evolutionary Network Architecture Search Framework with Adaptive
  Multimodal Fusion for Hand Gesture Recognition
Yizhang Xia Shihao Song Zhanglu Hou Junwen Xu Juan Zou Yuan Liu Shengxiang Yang
Hand gesture recognition (HGR) based on multimodal data has attracted
considerable attention owing to its great potential in applications. Various
manually designed multimodal deep networks have performed well in multimodal
HGR (MHGR), but most of existing algorithms require a lot of expert experience
and time-consuming manual trials. To address these issues, we propose an
evolutionary network architecture search framework with the adaptive multimodel
fusion (AMF-ENAS). Specifically, we design an encoding space that
simultaneously considers fusion positions and ratios of the multimodal data,
allowing for the automatic construction of multimodal networks with different
architectures through decoding. Additionally, we consider three input streams
corresponding to intra-modal surface electromyography (sEMG), intra-modal
accelerometer (ACC), and inter-modal sEMG-ACC. To automatically adapt to
various datasets, the ENAS framework is designed to automatically search a MHGR
network with appropriate fusion positions and ratios. To the best of our
knowledge, this is the first time that ENAS has been utilized in MHGR to tackle
issues related to the fusion position and ratio of multimodal data.
Experimental results demonstrate that AMF-ENAS achieves state-of-the-art
performance on the Ninapro DB2, DB3, and DB7 datasets.