Modelling 3D scenes from 2D images is a long-standing problem in computer vision with implications in, e.g., simulation and robotics. We propose pix2scene, a deep generative-based approach that implicitly models the geometric properties of a scene from images. Our method learns the depth and orientation of scene points visible in images. Our model can then predict the structure of a scene from various, previously unseen view points. It relies on a bi-directional adversarial learning mechanism to generate scene representations from a latent code, inferring the 3D representation of the underlying scene geometry. We showcase a novel differentiable renderer to train the 3D model in an end-to-end fashion, using only images. We demonstrate the generative ability of our model qualitatively on both a custom dataset and on ShapeNet. Finally, we evaluate the effectiveness of the learned 3D scene representation in supporting a 3D spatial reasoning. Understanding the 3-dimensional (3D) world from its 2-dimensional (2D) projections is a fundamental problem in computer vision with a broad range of application in robotics, simulation and design. Given that the majority natural scene data is available exclusively in the form of 2D images, the ability to directly infer knowledge about 3D structure from these images would be of great utility in scene understanding.Inferring the 3D structure from multiple images of a scene has been pursued extensively, such as in stereo or structure from motion tasks BID9 . Since most available natural image data informative about the real world comes with only a single view of a given scene, it is perhaps more important to explore the development of models which can infer the 3D structural properties from a single image. On the other hand, single image 3D recovery is an extremely challenging and heavily under constrained task. The system has to rely on prior knowledge and 2D visual cues such as textures, shadows or occlusions in order to provide hints to the 3D structure of the scene. Practically, building a machine learning model that learns to infer 3D structure from images requires either a strong inductive bias or supervision. While some have used the 3D ground truth as explicit supervision BID34 , in most cases of interest, such supervision will not be available. Consequently, our long term goal is to infer the 3D structure of realistic scenes from single images. In this paper we take a step towards this direction via a method of unsupervised learning of the 3D structure, directly from a single 2D image of each scene. Our method based on the adversarial learning framework BID6 and exploits a uniquely suitable 3D representation (i.e., surfels BID24 ) and a differentiable renderer.Most 3D reconstruction methods rely on representing 3D objects explicitly using either voxels BID26 BID36 or meshes BID14 BID31 . Explicit representations store all the rendering-relevant information from a given 3D space and are easily transferable, i.e., they can be loaded with any 3D modeling software and viewed from any angle. However, approaches using explicit representations typically scale very poorly (O(n 3 ) or require a sparse/discrete representation which can be challenging for deep learning methods. As a result, these representations have only been applied to the reconstruction of single objects. As an alternative we propose to learn an implicit 3D representation which produces only the 3D geometry which is directly relevant for a particular viewpoint. Our viewpoint-specific 3D geometry is captured Figure 1: Implicit vs explicit representations. Explicit voxel and mesh representations are viewpoint-independent and constitutes the complete scene. Our implicit surfel-based representation is viewpoint-dependent and it adapts the resolution to the viewpoint. The full scene is contained in a high-dimensional latent variable and only when the scene is to be rendered, the latent variable is serialized to surfels for a specific view.using camera facing surfels BID24 which are surface elements defined by its position, orientation and material properties. Given an image we can infer its implicit 3D representation and then recreate novel surfel representations of the underlying scene from unobserved viewpoints. In general, we note that in a 3D scene, only a small fraction of the entities are perceivable from the camera. As the camera moves, and the occluded regions become visible, our method then generates surfels for those newly unoccluded regions. Another advantage of this approach is that minimal number of primitives (surfels) are required to obtain a high-resolution image as the camera moves closer to a part of the scene. Moreover this representation fits well with image based convolutional architectures.Our model, Pix2Scene, is a deep generative-based approach for modelling the 3D structure of a scene directly from images. This model is unsupervised in the sense that it does not require 3D groundtruth or any other kind of image annotations. We base our model on Adversarially Learned Inference (ALI) approach BID5 . ALI extends the GAN BID6 framework by learning to infer the latent representation of a given image. In pix2scene the learned latent space embeds the 3D information of the underlying scene. The latent representation is mapped via a decoder network to a view-dependent 3D surface and then projected to image space by a differentiable renderer. The resulting image is then evaluated by an adversarial critic.While our long-term goal is to be able to infer the 3D structure of a real-world photograph, in this paper we experiment exclusively with synthetically-constructed scenes and adopt several simplifying assumptions. In particular, we assume that the world is piece-wise smooth and that for each input image the illumination, view and object materials are known.This work has the following main contributions, (1) we propose a novel unsupervised method for 3D understanding from a single image; (2) we propose a new implicit 3D representation based on view-space surfels; (3) we propose a surfel-based differentiable 3D renderer that can be used as a layer of a neural network; and (4) we propose 3D-IQTT a new 3D understanding evaluation benchmark. This task evaluates the model's ability to perform mental rotation by obtaining comprehensive understanding of underlying 3D structure. We also estimate the camera pose as part of the learnt latent variable for this particular task. In this paper we proposed a generative approach to learn 3D structural properties from single images in an unsupervised and implicit fashion. Our model receives an image of a scene with uniform material as input, estimates the depth of the scene points and then reconstructs the input scene. We also provided quantitative evidence that support our argument by introducing a novel IQ-task in a semi-supervised setup. We hope that this evaluation metric will be used as a standard benchmark to measure the 3D understanding capability of the models across different 3D representations. The main drawback of our current model is that it requires the knowledge of lighting and material properties. Future work will focus on tackling the more ambitious setting of learning complex materials and texture along with modelling the lighting properties of the scene.
pix2scene: a deep generative based approach for implicitly modelling the geometrical properties of a 3D scene from images Explores explaining scenes with surfels in a neural recognition model, and demonstrate results on image reconstruction, synthesis, and mental shape rotation. Authors introduce a method to create a 3D scene model given a 2D image and a camera pose using a self-superfised model