Generative adversarial networks (GANs) are a powerful framework for generative tasks. However, they are difficult to train and tend to miss modes of the true data generation process. Although GANs can learn a rich representation of the covered modes of the data in their latent space, the framework misses an inverse mapping from data to this latent space. We propose Invariant Encoding Generative Adversarial Networks (IVE-GANs), a novel GAN framework that introduces such a mapping for individual samples from the data by utilizing features in the data which are invariant to certain transformations. Since the model maps individual samples to the latent space, it naturally encourages the generator to cover all modes. We demonstrate the effectiveness of our approach in terms of generative performance and learning rich representations on several datasets including common benchmark image generation tasks. Generative Adversarial Networks (GANs) BID4 emerged as a powerful framework for training generative models in the recent years. GANs consist of two competing (adversarial) networks: a generative model that tries to capture the distribution of a given dataset to map from an arbitrary latent space (usually drawn from a multi-variate Gaussian) to new synthetic data points, and a discriminative model that tries to distinguish between samples from the generator and the true data. Iterative training of both models ideally results in a discriminator capturing features from the true data that the generator does not synthesize, while the generator learns to include these features in the generative process, until real and synthesized data are no longer distinguishable. Experiments by BID10 showed that a GAN can learn rich representation of the data in the latent space in which interpolations produce semantic variations and shifts in certain directions correspond to variations of specific features of the generated data. However, due to the lack of an inverse mapping from data to the latent space, GANs cannot be used to encode individual data points in the latent space BID2 . Moreover, although GANs show promising results in various tasks, such as the generation of realistic looking images BID9 BID0 BID11 or 3D objects BID13 , training a GAN in the aforementioned ideal way is difficult to set up and sensitive to hyper-parameter selection BID10 . Additionally, GANs tend to restrict themselves on generating only a few major modes of the true data distribution, since such a so-called mode collapse is not penalized in the GAN objective, while resulting in more realistic samples from these modes BID1 . Hence, the majority of the latent space only maps to a few regions in the target space resulting in poor representation of the true data. We propose a novel GAN framework, Invariant-Encoding Generative Adversarial Networks (IVE-GAN), which extends the classical GAN architecture by an additional encoding unit E to map samples from the true data x to the latent space z (compare FIG0 ). To encourage the encoder to learn a rich representation of the data in the latent space, the discriminator D is asked to distinguish between different predefined transformations T(x) of the input sample and generated samples G(E(x)) by taking the the original input as condition into account. While the discriminator has to learn what the different variations have in common with the original input, the encoder is forced to encode the necessary information in the latent space so that the generator can fool the discriminator by generating samples which are similar to the original samples. Since the discriminator is invariant to the predefined transformations, the encoder can ignore these variations in the input space and learn a rich and to such transformations invariant representation of the data. The variations of the generated samples are modeled by an additional latent vector z (drawn from a multi-variate Gaussian). Thus, the encoded samples condition the generator G(z , E(x)). Moreover, since the discriminator learns to distinguish between generated samples and variations of the original for each individual sample x in the data, the latent space can not collapse to a few modes of the true data distribution since it will be easy for the discriminator to distinguish generated samples from original ones if the mode is not covered by the generator. Thus, the proposed IVE-GAN learns a rich and to certain transformations invariant representation of a dataset and naturally encourages the generator to cover all modes of the data distribution.To generate novel samples, the generator G(z) can be fed with an arbitrary latent representation z ∼ P noise .In summary, we make the following contributions:• We derive a novel GAN framework for learning rich and transformation invariant representation of the data in the latent space.• We show that our GANs reproduce samples from a data distribution without mode collapsing issues.• We demonstrate robust GAN training across various data sets and showcase that our GAN produces very realistic samples. With this work we proposed a novel GAN framework that includes a encoding unit that maps data to a latent representation by utilizing features in the data which are invariant to certain transformations.We evaluate the proposed model on three different dataset and show the IVE-GAN can generate visually appealing images of high variance while learning a rich representation of the dataset also covering subtle features. B NETWORK ARCHITECTURE AND HYPERPARAMETERS
A noval GAN framework that utilizes transformation-invariant features to learn rich representations and strong generators. Proposes a modified GAN objective consisting of a classic GAN term and an invariant encoding term. This paper presents the IVE-GAN, a model that introduces en encoder to the Generative Adversarial Network framework.