HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable
  Sparse Observations
Peng Dai Yang Zhang Tao Liu Zhen Fan Tianyuan Du Zhuo Su Xiaozheng Zheng Zeming Li
It is especially challenging to achieve real-time human motion tracking on a
standalone VR Head-Mounted Display (HMD) such as Meta Quest and PICO. In this
paper, we propose HMD-Poser, the first unified approach to recover full-body
motions using scalable sparse observations from HMD and body-worn IMUs. In
particular, it can support a variety of input scenarios, such as HMD,
HMD+2IMUs, HMD+3IMUs, etc. The scalability of inputs may accommodate users'
choices for both high tracking accuracy and easy-to-wear. A lightweight
temporal-spatial feature learning network is proposed in HMD-Poser to guarantee
that the model runs in real-time on HMDs. Furthermore, HMD-Poser presents
online body shape estimation to improve the position accuracy of body joints.
Extensive experimental results on the challenging AMASS dataset show that
HMD-Poser achieves new state-of-the-art results in both accuracy and real-time
performance. We also build a new free-dancing motion dataset to evaluate
HMD-Poser's on-device performance and investigate the performance gap between
synthetic data and real-captured sensor data. Finally, we demonstrate our
HMD-Poser with a real-time Avatar-driving application on a commercial HMD. Our
code and free-dancing motion dataset are available
https://pico-ai-team.github.io/hmd-poser