HMD-NeMo: Online 3D Avatar Motion Generation From Sparse Observations
Sadegh Aliakbarian Fatemeh Saleh David Collier Pashmina Cameron Darren Cosker
Generating both plausible and accurate full body avatar motion is the key to
the quality of immersive experiences in mixed reality scenarios. Head-Mounted
Devices (HMDs) typically only provide a few input signals, such as head and
hands 6-DoF. Recently, different approaches achieved impressive performance in
generating full body motion given only head and hands signal. However, to the
best of our knowledge, all existing approaches rely on full hand visibility.
While this is the case when, e.g., using motion controllers, a considerable
proportion of mixed reality experiences do not involve motion controllers and
instead rely on egocentric hand tracking. This introduces the challenge of
partial hand visibility owing to the restricted field of view of the HMD. In
this paper, we propose the first unified approach, HMD-NeMo, that addresses
plausible and accurate full body motion generation even when the hands may be
only partially visible. HMD-NeMo is a lightweight neural network that predicts
the full body motion in an online and real-time fashion. At the heart of
HMD-NeMo is the spatio-temporal encoder with novel temporally adaptable mask
tokens that encourage plausible motion in the absence of hand observations. We
perform extensive analysis of the impact of different components in HMD-NeMo
and introduce a new state-of-the-art on AMASS dataset through our evaluation.