Component attention network for multimodal dance improvisation
  recognition
Jia Fu Jiarui Tan Wenjie Yin Sepideh Pashami Mårten Björkman
Dance improvisation is an active research topic in the arts. Motion analysis
of improvised dance can be challenging due to its unique dynamics. Data-driven
dance motion analysis, including recognition and generation, is often limited
to skeletal data. However, data of other modalities, such as audio, can be
recorded and benefit downstream tasks. This paper explores the application and
performance of multimodal fusion methods for human motion recognition in the
context of dance improvisation. We propose an attention-based model, component
attention network (CANet), for multimodal fusion on three levels: 1) feature
fusion with CANet, 2) model fusion with CANet and graph convolutional network
(GCN), and 3) late fusion with a voting strategy. We conduct thorough
experiments to analyze the impact of each modality in different fusion methods
and distinguish critical temporal or component features. We show that our
proposed model outperforms the two baseline methods, demonstrating its
potential for analyzing improvisation in dance.