Novel View Synthesis from a Single RGBD Image for Indoor Scenes
Congrui Hetang Yuping Wang
In this paper, we propose an approach for synthesizing novel view images from
a single RGBD (Red Green Blue-Depth) input. Novel view synthesis (NVS) is an
interesting computer vision task with extensive applications. Methods using
multiple images has been well-studied, exemplary ones include training
scene-specific Neural Radiance Fields (NeRF), or leveraging multi-view stereo
(MVS) and 3D rendering pipelines. However, both are either computationally
intensive or non-generalizable across different scenes, limiting their
practical value. Conversely, the depth information embedded in RGBD images
unlocks 3D potential from a singular view, simplifying NVS. The widespread
availability of compact, affordable stereo cameras, and even LiDARs in
contemporary devices like smartphones, makes capturing RGBD images more
accessible than ever. In our method, we convert an RGBD image into a point
cloud and render it from a different viewpoint, then formulate the NVS task
into an image translation problem. We leveraged generative adversarial networks
to style-transfer the rendered image, achieving a result similar to a
photograph taken from the new perspective. We explore both unsupervised
learning using CycleGAN and supervised learning with Pix2Pix, and demonstrate
the qualitative results. Our method circumvents the limitations of traditional
multi-image techniques, holding significant promise for practical, real-time
applications in NVS.