We capitalize on the natural compositional structure of images in order to learn object segmentation with weakly labeled images. The intuition behind our approach is that removing objects from images will yield natural images, however removing random patches will yield unnatural images. We leverage this signal to develop a generative model that decomposes an image into layers, and when all layers are combined, it reconstructs the input image. However, when a layer is removed, the model learns to produce a different image that still looks natural to an adversary, which is possible by removing objects. Experiments and visualizations suggest that this model automatically learns object segmentation on images labeled only by scene better than baselines. Visual recognition models demand large amounts of annotated data that is expensive to collect, and this cost is amplified for tasks that require densely labeled data, such as semantic segmentation. In this paper, we develop an approach where object segmentation emerges automatically for images only labeled by scene category.We capitalize on the natural compositional structure of images to learn object segmentation through counterfactual images. An image is counterfactual if it shows a real scene, except part of it has been removed or changed. To learn to segment, we train a model to generate counterfactual images such that they are perceptually realistic, a task the model can solve by removing objects and filling in the holes. For example, if you fully remove the bed from the scene in Figure 1 , the image is still realistic. However, if you only partially remove the bed, the image is not realistic anymore. We use this intuition to automatically learn object segmentation.We develop a stochastic layered model that decomposes an input image into several layers. We train this model so that when all layers are combined together in some order, it reconstructs the input image. However, we also train the model so that if we randomly permute the layers and remove a layer, the combination still appears perceptually real to an adversary. Consequently, the model learns a layered image decomposition that allows parts of the image to be removed. We show that the model automatically learns to isolate objects in different layers in order to make the output image still appear realistic, a signal we capitalize on for learning to segment.We present three main experiments to analyze this approach. Firstly, experiments show that our model learns to automatically segment images into objects for some scene categories, with only weakly labeled training data, and our approach outperforms several baselines. Secondly, we show that we use a small amount of densely labeled data with our approach to further improve performance. Finally, visualizations suggest that the model can generate the scene behind objects that it learns to segment, enabling us to remove pictures from a wall or take off the bed sheets.Our main contribution is to introduce a novel method for object segmentation on data only labeled by scene by capitalizing on natural compositional structures in images. While the focus of this paper is on images, the method is general and could be applied to other signals, such as audio. The remainder of this paper describes this contribution. Section 2 reviews related work. Section 3 present our method to auto-encode images with a layered decomposition, and shows how removing image regions is a useful signal for segmentation. Section 4 shows several experiments for semantic segmentation, and section 5 offers concluding remarks. We plan to release all code, data, and models.
Weakly-supervised image segmentation using compositional structure of images and generative models. This paper creates a layered representation in order to better learn segmentation from unlabeled images. This paper proposes a GAN-based generative model that decomposes images into multiple layers, where the objective of the GAN is to distinguish real images from images formed by combining the layers. This paper proposes a neural network architecture around the idea of layered scene composition