Learning Human Motion from Monocular Videos via Cross-Modal Manifold
  Alignment
Shuaiying Hou Hongyu Tao Junheng Fang Changqing Zou Hujun Bao Weiwei Xu
Learning 3D human motion from 2D inputs is a fundamental task in the realms
of computer vision and computer graphics. Many previous methods grapple with
this inherently ambiguous task by introducing motion priors into the learning
process. However, these approaches face difficulties in defining the complete
configurations of such priors or training a robust model. In this paper, we
present the Video-to-Motion Generator (VTM), which leverages motion priors
through cross-modal latent feature space alignment between 3D human motion and
2D inputs, namely videos and 2D keypoints. To reduce the complexity of modeling
motion priors, we model the motion data separately for the upper and lower body
parts. Additionally, we align the motion data with a scale-invariant virtual
skeleton to mitigate the interference of human skeleton variations to the
motion priors. Evaluated on AIST++, the VTM showcases state-of-the-art
performance in reconstructing 3D human motion from monocular videos. Notably,
our VTM exhibits the capabilities for generalization to unseen view angles and
in-the-wild videos.