3D Points Splatting for Real-Time Dynamic Hand Reconstruction
Zheheng Jiang Hossein Rahmani Sue Black Bryan M. Williams
We present 3D Points Splatting Hand Reconstruction (3D-PSHR), a real-time and
photo-realistic hand reconstruction approach. We propose a self-adaptive
canonical points upsampling strategy to achieve high-resolution hand geometry
representation. This is followed by a self-adaptive deformation that deforms
the hand from the canonical space to the target pose, adapting to the dynamic
changing of canonical points which, in contrast to the common practice of
subdividing the MANO model, offers greater flexibility and results in improved
geometry fitting. To model texture, we disentangle the appearance color into
the intrinsic albedo and pose-aware shading, which are learned through a
Context-Attention module. Moreover, our approach allows the geometric and the
appearance models to be trained simultaneously in an end-to-end manner. We
demonstrate that our method is capable of producing animatable, photorealistic
and relightable hand reconstructions using multiple datasets, including
monocular videos captured with handheld smartphones and large-scale multi-view
videos featuring various hand poses. We also demonstrate that our approach
achieves real-time rendering speeds while simultaneously maintaining superior
performance compared to existing state-of-the-art methods.