Gesture Recognition with mmWave Wi-Fi Access Points: Lessons Learned
Nabeel Nisar Bhat Rafael Berkvens Jeroen Famaey
In recent years, channel state information (CSI) at sub-6 GHz has been widely
exploited for Wi-Fi sensing, particularly for activity and gesture recognition.
In this work, we instead explore mmWave (60 GHz) Wi-Fi signals for gesture
recognition/pose estimation. Our focus is on the mmWave Wi-Fi signals so that
they can be used not only for high data rate communication but also for
improved sensing e.g., for extended reality (XR) applications. For this reason,
we extract spatial beam signal-to-noise ratios (SNRs) from the periodic beam
training employed by IEEE 802.11ad devices. We consider a set of 10
gestures/poses motivated by XR applications. We conduct experiments in two
environments and with three people.As a comparison, we also collect CSI from
IEEE 802.11ac devices. To extract features from the CSI and the beam SNR, we
leverage a deep neural network (DNN). The DNN classifier achieves promising
results on the beam SNR task with state-of-the-art 96.7% accuracy in a single
environment, even with a limited dataset. We also investigate the robustness of
the beam SNR against CSI across different environments. Our experiments reveal
that features from the CSI generalize without additional re-training, while
those from beam SNRs do not. Therefore, re-training is required in the latter
case.