In-Hand 3D Object Reconstruction from a Monocular RGB Video
Shijian Jiang Qi Ye Rengan Xie Yuchi Huo Xiang Li Yang Zhou Jiming Chen
Our work aims to reconstruct a 3D object that is held and rotated by a hand
in front of a static RGB camera. Previous methods that use implicit neural
representations to recover the geometry of a generic hand-held object from
multi-view images achieved compelling results in the visible part of the
object. However, these methods falter in accurately capturing the shape within
the hand-object contact region due to occlusion. In this paper, we propose a
novel method that deals with surface reconstruction under occlusion by
incorporating priors of 2D occlusion elucidation and physical contact
constraints. For the former, we introduce an object amodal completion network
to infer the 2D complete mask of objects under occlusion. To ensure the
accuracy and view consistency of the predicted 2D amodal masks, we devise a
joint optimization method for both amodal mask refinement and 3D
reconstruction. For the latter, we impose penetration and attraction
constraints on the local geometry in contact regions. We evaluate our approach
on HO3D and HOD datasets and demonstrate that it outperforms the
state-of-the-art methods in terms of reconstruction surface quality, with an
improvement of $52\%$ on HO3D and $20\%$ on HOD. Project webpage:
https://east-j.github.io/ihor.