DivaTrack: Diverse Bodies and Motions from Acceleration-Enhanced
  Three-Point Trackers
Dongseok Yang Jiho Kang Lingni Ma Joseph Greer Yuting Ye Sung-Hee Lee
Full-body avatar presence is crucial for immersive social and environmental
interactions in digital reality. However, current devices only provide three
six degrees of freedom (DOF) poses from the headset and two controllers (i.e.
three-point trackers). Because it is a highly under-constrained problem,
inferring full-body pose from these inputs is challenging, especially when
supporting the full range of body proportions and use cases represented by the
general population. In this paper, we propose a deep learning framework,
DivaTrack, which outperforms existing methods when applied to diverse body
sizes and activities. We augment the sparse three-point inputs with linear
accelerations from Inertial Measurement Units (IMU) to improve foot contact
prediction. We then condition the otherwise ambiguous lower-body pose with the
predictions of foot contact and upper-body pose in a two-stage model. We
further stabilize the inferred full-body pose in a wide range of configurations
by learning to blend predictions that are computed in two reference frames,
each of which is designed for different types of motions. We demonstrate the
effectiveness of our design on a large dataset that captures 22 subjects
performing challenging locomotion for three-point tracking, including lunges,
hula-hooping, and sitting. As shown in a live demo using the Meta VR headset
and Xsens IMUs, our method runs in real-time while accurately tracking a user's
motion when they perform a diverse set of movements.