Semantic-Human: Neural Rendering of Humans from Monocular Video with
  Human Parsing
Jie Zhang Pengcheng Shi Zaiwang Gu Yiyang Zhou Zhi Wang
The neural rendering of humans is a topic of great research significance.
However, previous works mostly focus on achieving photorealistic details,
neglecting the exploration of human parsing. Additionally, classical semantic
work are all limited in their ability to efficiently represent fine results in
complex motions. Human parsing is inherently related to radiance
reconstruction, as similar appearance and geometry often correspond to similar
semantic part. Furthermore, previous works often design a motion field that
maps from the observation space to the canonical space, while it tends to
exhibit either underfitting or overfitting, resulting in limited
generalization. In this paper, we present Semantic-Human, a novel method that
achieves both photorealistic details and viewpoint-consistent human parsing for
the neural rendering of humans. Specifically, we extend neural radiance fields
(NeRF) to jointly encode semantics, appearance and geometry to achieve accurate
2D semantic labels using noisy pseudo-label supervision. Leveraging the
inherent consistency and smoothness properties of NeRF, Semantic-Human achieves
consistent human parsing in both continuous and novel views. We also introduce
constraints derived from the SMPL surface for the motion field and
regularization for the recovered volumetric geometry. We have evaluated the
model using the ZJU-MoCap dataset, and the obtained highly competitive results
demonstrate the effectiveness of our proposed Semantic-Human. We also showcase
various compelling applications, including label denoising, label synthesis and
image editing, and empirically validate its advantageous properties.