P2I-NET: Mapping Camera Pose to Image via Adversarial Learning for New
  View Synthesis in Real Indoor Environments
Xujie Kang Kanglin Liu Jiang Duan Yuanhao Gong Guoping Qiu
Given a new $6DoF$ camera pose in an indoor environment, we study the
challenging problem of predicting the view from that pose based on a set of
reference RGBD views. Existing explicit or implicit 3D geometry construction
methods are computationally expensive while those based on learning have
predominantly focused on isolated views of object categories with regular
geometric structure. Differing from the traditional \textit{render-inpaint}
approach to new view synthesis in the real indoor environment, we propose a
conditional generative adversarial neural network (P2I-NET) to directly predict
the new view from the given pose. P2I-NET learns the conditional distribution
of the images of the environment for establishing the correspondence between
the camera pose and its view of the environment, and achieves this through a
number of innovative designs in its architecture and training lost function.
Two auxiliary discriminator constraints are introduced for enforcing the
consistency between the pose of the generated image and that of the
corresponding real world image in both the latent feature space and the real
world pose space. Additionally a deep convolutional neural network (CNN) is
introduced to further reinforce this consistency in the pixel space. We have
performed extensive new view synthesis experiments on real indoor datasets.
Results show that P2I-NET has superior performance against a number of NeRF
based strong baseline models. In particular, we show that P2I-NET is 40 to 100
times faster than these competitor techniques while synthesising similar
quality images. Furthermore, we contribute a new publicly available indoor
environment dataset containing 22 high resolution RGBD videos where each frame
also has accurate camera pose parameters.