TransHuman: A Transformer-based Human Representation for Generalizable
  Neural Human Rendering
Xiao Pan Zongxin Yang Jianxin Ma Chang Zhou Yi Yang
In this paper, we focus on the task of generalizable neural human rendering
which trains conditional Neural Radiance Fields (NeRF) from multi-view videos
of different characters. To handle the dynamic human motion, previous methods
have primarily used a SparseConvNet (SPC)-based human representation to process
the painted SMPL. However, such SPC-based representation i) optimizes under the
volatile observation space which leads to the pose-misalignment between
training and inference stages, and ii) lacks the global relationships among
human parts that is critical for handling the incomplete painted SMPL. Tackling
these issues, we present a brand-new framework named TransHuman, which learns
the painted SMPL under the canonical space and captures the global
relationships between human parts with transformers. Specifically, TransHuman
is mainly composed of Transformer-based Human Encoding (TransHE), Deformable
Partial Radiance Fields (DPaRF), and Fine-grained Detail Integration (FDI).
TransHE first processes the painted SMPL under the canonical space via
transformers for capturing the global relationships between human parts. Then,
DPaRF binds each output token with a deformable radiance field for encoding the
query point under the observation space. Finally, the FDI is employed to
further integrate fine-grained information from reference images. Extensive
experiments on ZJU-MoCap and H36M show that our TransHuman achieves a
significantly new state-of-the-art performance with high efficiency. Project
page: https://pansanity666.github.io/TransHuman/