OCHID-Fi: Occlusion-Robust Hand Pose Estimation in 3D via RF-Vision
Shujie Zhang Tianyue Zheng Zhe Chen Jingzhi Hu Abdelwahed Khamis Jiajun Liu Jun Luo
Hand Pose Estimation (HPE) is crucial to many applications, but conventional
cameras-based CM-HPE methods are completely subject to Line-of-Sight (LoS), as
cameras cannot capture occluded objects. In this paper, we propose to exploit
Radio-Frequency-Vision (RF-vision) capable of bypassing obstacles for achieving
occluded HPE, and we introduce OCHID-Fi as the first RF-HPE method with 3D pose
estimation capability. OCHID-Fi employs wideband RF sensors widely available on
smart devices (e.g., iPhones) to probe 3D human hand pose and extract their
skeletons behind obstacles. To overcome the challenge in labeling RF imaging
given its human incomprehensible nature, OCHID-Fi employs a cross-modality and
cross-domain training process. It uses a pre-trained CM-HPE network and a
synchronized CM/RF dataset, to guide the training of its complex-valued RF-HPE
network under LoS conditions. It further transfers knowledge learned from
labeled LoS domain to unlabeled occluded domain via adversarial learning,
enabling OCHID-Fi to generalize to unseen occluded scenarios. Experimental
results demonstrate the superiority of OCHID-Fi: it achieves comparable
accuracy to CM-HPE under normal conditions while maintaining such accuracy even
in occluded scenarios, with empirical evidence for its generalizability to new
domains.