MESEN: Exploit Multimodal Data to Design Unimodal Human Activity
  Recognition with Few Labels
Lilin Xu Chaojie Gu Rui Tan Shibo He Jiming Chen
Human activity recognition (HAR) will be an essential function of various
emerging applications. However, HAR typically encounters challenges related to
modality limitations and label scarcity, leading to an application gap between
current solutions and real-world requirements. In this work, we propose MESEN,
a multimodal-empowered unimodal sensing framework, to utilize unlabeled
multimodal data available during the HAR model design phase for unimodal HAR
enhancement during the deployment phase. From a study on the impact of
supervised multimodal fusion on unimodal feature extraction, MESEN is designed
to feature a multi-task mechanism during the multimodal-aided pre-training
stage. With the proposed mechanism integrating cross-modal feature contrastive
learning and multimodal pseudo-classification aligning, MESEN exploits
unlabeled multimodal data to extract effective unimodal features for each
modality. Subsequently, MESEN can adapt to downstream unimodal HAR with only a
few labeled samples. Extensive experiments on eight public multimodal datasets
demonstrate that MESEN achieves significant performance improvements over
state-of-the-art baselines in enhancing unimodal HAR by exploiting multimodal
data.