We propose a new algorithm for training generative adversarial networks to jointly learn latent codes for both identities (e.g. individual humans) and observations (e.g. specific photographs). In practice, this means that by fixing the identity portion of latent codes, we can generate diverse images of the same subject, and by fixing the observation portion we can traverse the manifold of subjects while maintaining contingent aspects such as lighting and pose. Our algorithm features a pairwise training scheme in which each sample from the generator consists of two images with a common identity code. Corresponding samples from the real dataset consist of two distinct photographs of the same subject. In order to fool the discriminator, the generator must produce images that are both photorealistic, distinct, and appear to depict the same person. We augment both the DCGAN and BEGAN approaches with Siamese discriminators to accommodate pairwise training. Experiments with human judges and an off-the-shelf face verification system demonstrate our algorithm’s ability to generate convincing, identity-matched photographs. In many domains, a suitable generative process might consist of several stages. To generate a photograph of a product, we might wish to first sample from the space of products, and then from the space of photographs of that product. Given such disentangled representations in a multistage generative process, an online retailer might diversify its catalog, depicting products in a wider variety of settings. A retailer could also flip the process, imagining new products in a fixed setting. Datasets for such domains often contain many labeled identities with fewer observations of each (e.g. a collection of face portraits with thousands of people and ten photos of each). While we may know the identity of the subject in each photograph, we may not know the contingent aspects of the observation (such as lighting, pose and background). This kind of data is ubiquitous; given a set of commonalities, we might want to incorporate this structure into our latent representations.Generative adversarial networks (GANs) learn mappings from latent codes z in some low-dimensional space Z to points in the space of natural data X BID9 . They achieve this power through an adversarial training scheme pitting a generative model G : Z → X against a discriminative model D : X → [0, 1] in a minimax game. While GANs are popular, owing to their ability to generate high-fidelity images, they do not, in their original form, explicitly disentangle the latent factors according to known commonalities.In this paper, we propose Semantically Decomposed GANs (SD-GANs), which encourage a specified portion of the latent space to correspond to a known source of variation.1,2 The technique Figure 1 : Generated samples from SD-BEGAN. Each of the four rows has the same identity code z I and each of the fourteen columns has the same observation code z O .decomposes the latent code Z into one portion Z I corresponding to identity, and the remaining portion Z O corresponding to the other contingent aspects of observations. SD-GANs learn through a pairwise training scheme in which each sample from the real dataset consists of two distinct images with a common identity. Each sample from the generator consists of a pair of images with common z I ∈ Z I but differing z O ∈ Z O . In order to fool the discriminator, the generator must not only produce diverse and photorealistic images, but also images that depict the same identity when z I is fixed. For SD-GANs, we modify the discriminator so that it can determine whether a pair of samples constitutes a match.As a case study, we experiment with a dataset of face photographs, demonstrating that SD-GANs can generate contrasting images of the same subject ( Figure 1 ; interactive web demo in footnote on previous page). The generator learns that certain properties are free to vary across observations but not identity. For example, SD-GANs learn that pose, facial expression, hirsuteness, grayscale vs. color, and lighting can all vary across different photographs of the same individual. On the other hand, the aspects that are more salient for facial verification remain consistent as we vary the observation code z O . We also train SD-GANs on a dataset of product images, containing multiple photographs of each product from various perspectives FIG2 ).We demonstrate that SD-GANs trained on faces generate stylistically-contrasting, identity-matched image pairs that human annotators and a state-of-the-art face verification algorithm recognize as depicting the same subject. On measures of identity coherence and image diversity, SD-GANs perform comparably to a recent conditional GAN method (Odena et al., 2017) ; SD-GANs can also imagine new identities, while conditional GANs are limited to generating existing identities from the training data. Our evaluation demonstrates that SD-GANs can disentangle those factors of variation corresponding to identity from the rest. Moreover, with SD-GANs we can sample never-before-seen identities, a benefit not shared by conditional GANs. In FIG1 , we demonstrate that by varying the observation vector z O , SD-GANs can change the color of clothing, add or remove sunnies, or change facial pose. They can also perturb the lighting, color saturation, and contrast of an image, all while keeping the apparent identity fixed. We note, subjectively, that samples from SD-DCGAN tend to appear less photorealistic than those from SD-BEGAN. Given a generator trained with SD-GAN, we can independently interpolate along the identity and observation manifolds ( FIG4 ).On the shoe dataset, we find that the SD-DCGAN model produces convincing results. As desired, manipulating z I while keeping z O fixed yields distinct shoes in consistent poses FIG2 . The identity code z I appears to capture the broad categories of shoes (sneakers, flip-flops, boots, etc.) . Surprisingly , neither original BEGAN nor SD-BEGAN can produce diverse shoe images (Appendix G).In this paper , we presented SD-GANs, a new algorithm capable of disentangling factors of variation according to known commonalities. We see several promising directions for future work. One logical extension is to disentangle latent factors corresponding to more than one known commonality. We also plan to apply our approach in other domains such as identity-conditioned speech synthesis. We estimate latent vectors for unseen images and demonstrate that the disentangled representations of SD-GANs can be used to depict the estimated identity with different contingent factors. In order to find a latent vectorẑ such that G(ẑ) (pretrained G) is similar to an unseen image x, we can minimize the distance between x and G(ẑ): minẑ ||G(ẑ) − x|| In FIG5 , we depict estimation and linear interpolation across both subspaces for two pairs of images using SD-BEGAN. We also display the corresponding source images being estimated. For both pairs,ẑ I (identity) is consistent in each row andẑ O (observation) is consistent in each column.
SD-GANs disentangle latent codes according to known commonalities in a dataset (e.g. photographs depicting the same person). This paper investigates the problem of controlled image generation and proposes an algorithm that produces a pair of images with the same identity. This paper proposes, SD-GAN, a method of training GANs to disentangle the identity and non-identity information in the latent vector input Z.