MaskFi: Unsupervised Learning of WiFi and Vision Representations for
  Multimodal Human Activity Recognition
Jianfei Yang Shijie Tang Yuecong Xu Yunjiao Zhou Lihua Xie
Human activity recognition (HAR) has been playing an increasingly important
role in various domains such as healthcare, security monitoring, and metaverse
gaming. Though numerous HAR methods based on computer vision have been
developed to show prominent performance, they still suffer from poor robustness
in adverse visual conditions in particular low illumination, which motivates
WiFi-based HAR to serve as a good complementary modality. Existing solutions
using WiFi and vision modalities rely on massive labeled data that are very
cumbersome to collect. In this paper, we propose a novel unsupervised
multimodal HAR solution, MaskFi, that leverages only unlabeled video and WiFi
activity data for model training. We propose a new algorithm, masked
WiFi-vision modeling (MI2M), that enables the model to learn cross-modal and
single-modal features by predicting the masked sections in representation
learning. Benefiting from our unsupervised learning procedure, the network
requires only a small amount of annotated data for finetuning and can adapt to
the new environment with better performance. We conduct extensive experiments
on two WiFi-vision datasets collected in-house, and our method achieves human
activity recognition and human identification in terms of both robustness and
accuracy.