Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches
  and a Head-Mounted Camera
Jiye Lee Hanbyul Joo
We present a lightweight and affordable motion capture method based on two
smartwatches and a head-mounted camera. In contrast to the existing approaches
that use six or more expert-level IMU devices, our approach is much more
cost-effective and convenient. Our method can make wearable motion capture
accessible to everyone everywhere, enabling 3D full-body motion capture in
diverse environments. As a key idea to overcome the extreme sparsity and
ambiguities of sensor inputs, we integrate 6D head poses obtained from the
head-mounted cameras for motion estimation. To enable capture in expansive
indoor and outdoor scenes, we propose an algorithm to track and update floor
level changes to define head poses, coupled with a multi-stage
Transformer-based regression module. We also introduce novel strategies
leveraging visual cues of egocentric images to further enhance the motion
capture quality while reducing ambiguities. We demonstrate the performance of
our method on various challenging scenarios, including complex outdoor
environments and everyday motions including object interactions and social
interactions among multiple individuals.