Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs). However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves significant changes in shape, e.g., translating pants to skirts in fashion images. To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN), that incorporates the instance information (e.g., object segmentation masks) and improves multi-instance transfiguration. The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances. To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances. We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances. Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases. Code and results are available in https://github.com/sangwoomo/instagan Cross-domain generation arises in many machine learning tasks, including neural machine translation BID2 BID21 , image synthesis BID33 BID48 , text style transfer BID34 , and video generation BID3 BID39 BID6 . In particular, the unpaired (or unsupervised) image-to-image translation has achieved an impressive progress based on variants of generative adversarial networks (GANs) BID27 BID7 BID0 BID23 , and has also drawn considerable attention due to its practical applications including colorization , super-resolution BID22 , semantic manipulation BID40 , and domain adaptation BID5 BID35 BID15 . Previous methods on this line of research, however, often fail on challenging tasks, in particular, when the translation task involves significant changes in shape of instances or the images to translate contains multiple target instances BID10 . Our goal is to extend image-to-image translation towards such challenging tasks, which can strengthen its applicability up to the next level, e.g., changing pants to skirts in fashion images for a customer to decide which one is better to buy. To this end, we propose a novel method that incorporates the instance information of multiple target objectsin the framework of generative adversarial networks (GAN); hence we called it instance-aware GAN (InstaGAN) . In this work, we use the object segmentation masks for instance information, which may be a good representation for instance shapes, as it contains object boundaries while ignoring other details such as color. Using the information, our method shows impressive results for multi-instance transfiguration tasks, as shown in FIG0 .Our main contribution is three-fold: an instance-augmented neural architecture, a context preserving loss, and a sequential mini-batch inference/training technique. First , we propose a neural network architecture that translates both an image and the corresponding set of instance attributes. Our architecture can translate an arbitrary number of instance attributes conditioned by the input, and is designed to be permutation-invariant to the order of instances. Second , we propose a context preserv- ), and our proposed method, InstaGAN. Our method shows better results for multi-instance transfiguration problems.ing loss that encourages the network to focus on target instances in translation and learn an identity function outside of them. Namely, it aims at preserving the background context while transforming the target instances. Finally, we propose a sequential mini-batch inference/training technique, i.e., translating the mini-batches of instance attributes sequentially, instead of doing the entire set at once. It allows to handle a large number of instance attributes with a limited GPU memory, and thus enhances the network to generalize better for images with many instances. Furthermore, it improves the translation quality of images with even a few instances because it acts as data augmentation during training by producing multiple intermediate samples. All the aforementioned contributions are dedicated to how to incorporates the instance information (e.g., segmentation masks) for image-to-image translation. However, we believe that our approach is applicable to numerous other cross-domain generation tasks where set-structured side information is available.To the best of our knowledge, we are the first to report image-to-image translation results for multiinstance transfiguration tasks. A few number of recent methods BID27 BID10 show some transfiguration results but only for images with a single instance often in a clear background. Unlike the previous results in a simple setting, our focus is on the harmony of instances naturally rendered with the background. On the other hand, CycleGAN show some results for multi-instance cases, but report only a limited performance for transfiguration tasks. At a high level, the significance of our work is also on discovering that the instance information is effective for shape-transforming image-to-image translation, which we think would be influential to other related research in the future. Mask contrast-GAN and Attention-GAN (Mejjati et al., 2018) use segmentation masks or predicted attentions, but only to attach the background to the (translated) cropped instances. They do not allow to transform the shapes of the instances. To the contrary, our method learns how to preserve the background by optimizing the context preserving loss, thus facilitating the shape transformation. We have proposed a novel method incorporating the set of instance attributes for image-to-image translation. The experiments on different datasets have shown successful image-to-image translation on the challenging tasks of multi-instance transfiguration, including new tasks, e.g., translating jeans to skirt in fashion images. We remark that our ideas utilizing the set-structured side information have potential to be applied to other cross-domain generations tasks, e.g., neural machine translation or video generation. Investigating new tasks and new information could be an interesting research direction in the future.
We propose a novel method to incorporate the set of instance attributes for image-to-image translation. This paper proposes a method -- InstaGAN -- which builds on CycleGAN by taking into account instance information in the form of per-instance segmentation masks, with results that compare favorably to CycleGAN and other baselines.  Proposes to add instance-aware segmentation masks for the problem of unpaired image-to-image translation.