Novel-view Synthesis and Pose Estimation for Hand-Object Interaction
  from Sparse Views
Wentian Qu Zhaopeng Cui Yinda Zhang Chenyu Meng Cuixia Ma Xiaoming Deng Hongan Wang
Hand-object interaction understanding and the barely addressed novel view
synthesis are highly desired in the immersive communication, whereas it is
challenging due to the high deformation of hand and heavy occlusions between
hand and object. In this paper, we propose a neural rendering and pose
estimation system for hand-object interaction from sparse views, which can also
enable 3D hand-object interaction editing. We share the inspiration from recent
scene understanding work that shows a scene specific model built beforehand can
significantly improve and unblock vision tasks especially when inputs are
sparse, and extend it to the dynamic hand-object interaction scenario and
propose to solve the problem in two stages. We first learn the shape and
appearance prior knowledge of hands and objects separately with the neural
representation at the offline stage. During the online stage, we design a
rendering-based joint model fitting framework to understand the dynamic
hand-object interaction with the pre-built hand and object models as well as
interaction priors, which thereby overcomes penetration and separation issues
between hand and object and also enables novel view synthesis. In order to get
stable contact during the hand-object interaction process in a sequence, we
propose a stable contact loss to make the contact region to be consistent.
Experiments demonstrate that our method outperforms the state-of-the-art
methods. Code and dataset are available in project webpage
https://iscas3dv.github.io/HO-NeRF.