We propose a novel generative model architecture designed to learn representations for images that factor out a single attribute from the rest of the representation. A single object may have many attributes which when altered do not change the identity of the object itself. Consider the human face; the identity of a particular person is independent of whether or not they happen to be wearing glasses. The attribute of wearing glasses can be changed without changing the identity of the person. However, the ability to manipulate and alter image attributes without altering the object identity is not a trivial task. Here, we are interested in learning a representation of the image that separates the identity of an object (such as a human face) from an attribute (such as 'wearing glasses'). We demonstrate the success of our factorization approach by using the learned representation to synthesize the same face with and without a chosen attribute. We refer to this specific synthesis process as image attribute manipulation. We further demonstrate that our model achieves competitive scores, with state of the art, on a facial attribute classification task. Latent space generative models, such as generative adversarial networks (GANs) BID11 BID27 and variational autoencoders (VAEs) BID28 BID14 , learn a mapping from a latent encoding space to a data space, for example, the space of natural images. It has been shown that the latent space learned by these models is often organized in a near-linear fashion BID27 BID14 , whereby neighbouring points in latent space map to similar images in data space. Certain "directions" in latent space correspond to changes in the intensity of certain attributes. In the context of faces, for example, directions in latent space would correspond to the extent to which someone is smiling. This may be useful for image synthesis where one can use the latent space to develop new design concepts BID9 Zhu et al., 2016) , edit an existing image (Zhu et al., 2016) or synthesize avatars BID35 BID32 . This is because semantically meaningful changes may be made to images by manipulating the latent space BID27 Zhu et al., 2016; BID17 .One avenue of research for latent space generative models has been class conditional image synthesis BID25 BID24 , where an image of a particular object category is synthesized. Often , object categories may be sub-divided into fine-grain subcategories. For example, the category "dog" may be split into further sub-categories of different dog breeds. Work by BID3 propose latent space generative models for synthesizing images from fine-grained categories, in particular for synthesizing different celebrities' faces conditional on the identity of the celebrity.Rather than considering fine-grain categories, we propose to take steps towards solving the different, but related problem of image attribute manipulation. To solve this problem we want to be able to synthesize images and only change one element or attribute of its content. For example , if we are synthesizing faces we would like to edit whether or not a person is smiling. This is a different problem to fine-grain synthesis; we want to be able to synthesize two faces that are similar, with only a single chosen attribute changed, rather than synthesizing two different faces. The need to synthesis two faces that are similar makes the problem of image attribute manipulation more difficult than the fine-grain image synthesis problem; we need to learn a latent space representation that separates an object category from its attributes.In this paper, we propose a new model that learns a factored representation for faces, separating attribute information from the rest of the facial representation. We apply our model to the CelebA BID21 dataset of faces and control several facial attributes.Our contributions are as follows:1. Our core contribution is the novel cost function for training a VAE encoder to learn a latent representation which factorizes binary facial attribute information from a continuous identity representation (Section 3.2). 2. We provide an extensive quantitative analysis of the contributions of each of the many loss components in our model (Section 4.2). 3. We obtain classification scores that are competitive with state of the art (Zhuang et al., 2018) using the classifier that is already incorporated into the encoder of the VAE (Section 4.3). 4. We provide qualitative results demonstrating that our latent variable, generative model may be used to successfully edit the 'Smiling' attribute in more than 90% of the test cases (Section 4.4). 5. We discuss and clarify the distinction between conditional image synthesis and image attribute editing (Section 5). 6. We present code to reproduce experiments shown in this paper: (provided after review). We have proposed a novel perspective and approach to learning representations of images which subsequently allows elements, or attributes, of the image to be modified. We have demonstrated our approach on images of the human face, however, the method is generalisable to other objects. We modelled a human face in two parts, with a continuous latent vector that captures the identity of a person and a binary unit vector that captures a facial attribute, such as whether or not a person is smiling. By modelling an image with two separate representations, one for the object and the other for the object's attribute, we are able to change attributes without affecting the identity of the object. To learn this factored representation we have proposed a novel model aptly named Information Factorization conditional VAE-GAN. The model encourages the attribute information to be factored out of the identity representation via an adversarial learning process. Crucially, the representation learned by our model both captures identity faithfully and facilitates accurate and easy attribute editing without affecting identity. We have demonstrated that our model performs better than pre-existing models intended for category conditional image synthesis (Section 4.4), and have performed a detailed ablation study TAB0 which confirms the importance and relevance of our proposed method. Indeed, our model is highly effective as a classifier, achieving state of the art accuracy on facial attribute classification for several attributes (Figure 2 ). Our approach to learning factored representations for images is both a novel and important contribution to the general field of representation learning.
Learn representations for images that factor out a single attribute. This paper builds on Conditional VAE GANs to allow attribute manipulation in the synthesis process. This paper proposes a generative model to learn the representation which can separate the identity of an object from an attribute, and extends the autoencoder adversarial by adding an auxiliary network.