Recent Trends in 3D Reconstruction of General Non-Rigid Scenes
Raza Yunus Jan Eric Lenssen Michael Niemeyer Yiyi Liao Christian Rupprecht Christian Theobalt Gerard Pons-Moll Jia-Bin Huang Vladislav Golyanik Eddy Ilg
Reconstructing models of the real world, including 3D geometry, appearance,
and motion of real scenes, is essential for computer graphics and computer
vision. It enables the synthesizing of photorealistic novel views, useful for
the movie industry and AR/VR applications. It also facilitates the content
creation necessary in computer games and AR/VR by avoiding laborious manual
design processes. Further, such models are fundamental for intelligent
computing systems that need to interpret real-world scenes and actions to act
and interact safely with the human world. Notably, the world surrounding us is
dynamic, and reconstructing models of dynamic, non-rigidly moving scenes is a
severely underconstrained and challenging problem. This state-of-the-art report
(STAR) offers the reader a comprehensive summary of state-of-the-art techniques
with monocular and multi-view inputs such as data from RGB and RGB-D sensors,
among others, conveying an understanding of different approaches, their
potential applications, and promising further research directions. The report
covers 3D reconstruction of general non-rigid scenes and further addresses the
techniques for scene decomposition, editing and controlling, and generalizable
and generative modeling. More specifically, we first review the common and
fundamental concepts necessary to understand and navigate the field and then
discuss the state-of-the-art techniques by reviewing recent approaches that use
traditional and machine-learning-based neural representations, including a
discussion on the newly enabled applications. The STAR is concluded with a
discussion of the remaining limitations and open challenges.