Most deep latent factor models choose simple priors for simplicity, tractability
 or not knowing what prior to use. Recent studies show that the choice of
 the prior may have a profound effect on the expressiveness of the model,
 especially when its generative network has limited capacity. In this paper, we propose to learn a proper prior from data for adversarial autoencoders
 (AAEs). We introduce the notion of code generators to transform manually selected
 simple priors into ones that can better characterize the data distribution. Experimental results show that the proposed model can generate better image quality and learn better disentangled representations than
 AAEs in both supervised and unsupervised settings. Lastly, we present its
 ability to do cross-domain translation in a  text-to-image synthesis task. Deep latent factor models, such as variational autoencoders (VAEs) and adversarial autoencoders (AAEs), are becoming increasingly popular in various tasks, such as image generation BID6 , unsupervised clustering BID2 BID7 , and cross-domain translation BID10 . These models involve specifying a prior distribution over latent variables and defining a deep generative network (i.e., the decoder) that maps latent variables to data space in stochastic or deterministic fashion. Training such deep models usually requires learning a recognition network (i.e., the encoder) regularized by the prior.Traditionally, a simple prior, such as the standard normal distribution BID5 , is used for tractability, simplicity, or not knowing what prior to use. It is hoped that this simple prior will be transformed somewhere in the deep generative network into a form suitable for characterizing the data distribution. While this might hold true when the generative network has enough capacity, applying the standard normal prior often results in over-regularized models with only few active latent dimensions BID0 .Some recent works BID4 BID3 BID9 suggest that the choice of the prior may have a profound impact on the expressiveness of the model. As an example, in learning the VAE with a simple encoder and decoder, BID4 conjecture that multimodal priors can achieve a higher variational lower bound on the data loglikelihood than is possible with the standard normal prior. BID9 confirm the truth of this conjecture by showing that their multimodal prior, a mixture of the variational posteriors, consistently outperforms simple priors on a number of datasets in terms of maximizing the data log-likelihood. Taking one step further, BID3 learn a tree-structured nonparametric Bayesian prior for capturing the hierarchy of semantics presented in the data. All these priors are learned under the VAE framework following the principle of maximum likelihood.Along a similar line of thinking, we propose in this paper the notion of code generators for learning a prior from data for AAE. The objective is to learn a code generator network to transform a simple prior into one that, together with the generative network, can better characterize the data distribution. To this end, we generalize the framework of AAE in several significant ways:• We replace the simple prior with a learned prior by training the code generator to output latent variables that will minimize an adversarial loss in data space. • We employ a learned similarity metric BID6 in place of the default squared error in data space for training the autoencoder.• We maximize the mutual information between part of the code generator input and the decoder output for supervised and unsupervised training using a variational technique introduced in InfoGAN BID1 .Extensive experiments confirm its effectiveness of generating better quality images and learning better disentangled representations than AAE in both supervised and unsupervised settings, particularly on complicated datasets. In addition, to the best of our knowledge, this is one of the first few works that attempt to introduce a learned prior for AAE.The remainder of this paper is organized as follows: Section 2 reviews the background and related works. Section 3 presents the implementation details and the training process of the proposed code generator. Section 4 compares its performance with AAE in image generation and disentanglement tasks. Lastly, we conclude this paper with remarks on future work. In this paper, we propose to learn a proper prior from data for AAE. Built on the foundation of AAE, we introduce a code generator to transform the manually selected simple prior into one that can better fit the data distribution. We develop a training process that allows to learn both the autoencoder and the code generator simultaneously. We demonstrate its superior performance over AAE in image generation and learning disentangled representations in supervised and unsupervised settings. We also show its ability to do cross-domain translation. Mode collapse and training instability are two major issues to be further investigated in future work. Figure 14: Generated images in accordance with the varying color attribute in the text description "The flower is pink in color and has petals that are rounded in shape and ruffled." From left to right, the color attribute is set to pink, red, yellow, orange, purple, blue, white, green, and black, respectively. Note that there is no green or black flower in the dataset. Input latent code ∈ R code size 3 x 3 conv. 64 RELU stride 2 pad 1 4 x 4 upconv. 512 BN. RELU stride 1 3 x 3 residual blcok 64 4 x 4 up sampling residual block 256 stride 2 3 x 3 down sampling residual blcok 128 stride 2 4 x 4 up sampling residual block 128 stride 2 3 x 3 down sampling residual blcok 256 stride 2 4 x 4 up sampling residual block 64 stride 2 3 x 3 down sampling residual block 512 stride 2 3 x 3 conv. image channels Tanh 4 x 4 avg. pooling stride 1 FC. 2 x code size BN. RELU FC. code size Linear Input feature map FC. 2 x noise size BN. RELU 3 x 3 conv. out channels RELU stride 2 pad 1 FC. latent code size BN. Linear 3 x 3 conv. out channels RELU stride 1 pad 1 skip connection output = input + residual RELU Table 4
Learning Priors for Adversarial Autoencoders Proposes a simple extension of adversarial auto-encoders for conditional image generation. Focuses on adversarial autoencoders and introduces a code generator network to transform a simple prior into one that together with the generator can better fit the data distribution