360° Volumetric Portrait Avatar
Jalees Nehvi Berna Kabadayi Julien Valentin Justus Thies
We propose 360{\deg} Volumetric Portrait (3VP) Avatar, a novel method for
reconstructing 360{\deg} photo-realistic portrait avatars of human subjects
solely based on monocular video inputs. State-of-the-art monocular avatar
reconstruction methods rely on stable facial performance capturing. However,
the common usage of 3DMM-based facial tracking has its limits; side-views can
hardly be captured and it fails, especially, for back-views, as required inputs
like facial landmarks or human parsing masks are missing. This results in
incomplete avatar reconstructions that only cover the frontal hemisphere. In
contrast to this, we propose a template-based tracking of the torso, head and
facial expressions which allows us to cover the appearance of a human subject
from all sides. Thus, given a sequence of a subject that is rotating in front
of a single camera, we train a neural volumetric representation based on neural
radiance fields. A key challenge to construct this representation is the
modeling of appearance changes, especially, in the mouth region (i.e., lips and
teeth). We, therefore, propose a deformation-field-based blend basis which
allows us to interpolate between different appearance states. We evaluate our
approach on captured real-world data and compare against state-of-the-art
monocular reconstruction methods. In contrast to those, our method is the first
monocular technique that reconstructs an entire 360{\deg} avatar.