3D Visibility-aware Generalizable Neural Radiance Fields for Interacting
  Hands
Xuan Huang Hanhui Li Zejun Yang Zhisheng Wang Xiaodan Liang
Neural radiance fields (NeRFs) are promising 3D representations for scenes,
objects, and humans. However, most existing methods require multi-view inputs
and per-scene training, which limits their real-life applications. Moreover,
current methods focus on single-subject cases, leaving scenes of interacting
hands that involve severe inter-hand occlusions and challenging view variations
remain unsolved. To tackle these issues, this paper proposes a generalizable
visibility-aware NeRF (VA-NeRF) framework for interacting hands. Specifically,
given an image of interacting hands as input, our VA-NeRF first obtains a
mesh-based representation of hands and extracts their corresponding geometric
and textural features. Subsequently, a feature fusion module that exploits the
visibility of query points and mesh vertices is introduced to adaptively merge
features of both hands, enabling the recovery of features in unseen areas.
Additionally, our VA-NeRF is optimized together with a novel discriminator
within an adversarial learning paradigm. In contrast to conventional
discriminators that predict a single real/fake label for the synthesized image,
the proposed discriminator generates a pixel-wise visibility map, providing
fine-grained supervision for unseen areas and encouraging the VA-NeRF to
improve the visual quality of synthesized images. Experiments on the
Interhand2.6M dataset demonstrate that our proposed VA-NeRF outperforms
conventional NeRFs significantly. Project Page:
\url{https://github.com/XuanHuang0/VANeRF}.