fMPI: Fast Novel View Synthesis in the Wild with Layered Scene
  Representations
Jonas Kohler Nicolas Griffiths Sanchez Luca Cavalli Catherine Herold Albert Pumarola Alberto Garcia Garcia Ali Thabet
In this study, we propose two novel input processing paradigms for novel view
synthesis (NVS) methods based on layered scene representations that
significantly improve their runtime without compromising quality. Our approach
identifies and mitigates the two most time-consuming aspects of traditional
pipelines: building and processing the so-called plane sweep volume (PSV),
which is a high-dimensional tensor of planar re-projections of the input camera
views. In particular, we propose processing this tensor in parallel groups for
improved compute efficiency as well as super-sampling adjacent input planes to
generate denser, and hence more accurate scene representation. The proposed
enhancements offer significant flexibility, allowing for a balance between
performance and speed, thus making substantial steps toward real-time
applications. Furthermore, they are very general in the sense that any
PSV-based method can make use of them, including methods that employ multiplane
images, multisphere images, and layered depth images. In a comprehensive set of
experiments, we demonstrate that our proposed paradigms enable the design of an
NVS method that achieves state-of-the-art on public benchmarks while being up
to $50x$ faster than existing state-of-the-art methods. It also beats the
current forerunner in terms of speed by over $3x$, while achieving
significantly better rendering quality.