Modern generative models are usually designed to match target distributions directly in the data space, where the intrinsic dimensionality of data can be much lower than the ambient dimensionality. We argue that this discrepancy may contribute to the difficulties in training generative models. We therefore propose to map both the generated and target distributions to the latent space using the encoder of a standard autoencoder, and train the generator (or decoder) to match the target distribution in the latent space. The resulting method, perceptual generative autoencoder (PGA), is then incorporated with maximum likelihood or variational autoencoder (VAE) objective to train the generative model. With maximum likelihood, PGA generalizes the idea of reversible generative models to unrestricted neural network architectures and arbitrary latent dimensionalities. When combined with VAE, PGA can generate sharper samples than vanilla VAE. Recent years have witnessed great interest in generative models, mainly due to the success of generative adversarial networks (GANs) BID7 BID12 BID1 . Despite the prevalence, the adversarial nature of GANs can lead to a number of challenges, such as unstable training dynamics and mode collapse. Since the advent of GANs, substantial efforts have been devoted to addressing these challenges BID21 BID9 BID19 , while non-adversarial approaches that are free of these issues have also gained attention. Examples include variational autoencoders (VAEs) BID13 , reversible generative models BID5 BID14 , and Wasserstein autoencoders (WAEs) BID22 .However , non-adversarial approaches often have significant limitations. For instance , VAEs tend to generate blurry samples, while reversible generative models require restricted neural network architectures or solving neural differential equations BID8 . Furthermore , to use the change of variable formula, the latent space of a reversible model must have the same dimensionality as the data space, which is unreasonable considering that real-world, high-dimensional data (e.g., images) tends to lie on low-dimensional manifolds, and thus results in redundant latent dimensions and variability. Intriguingly , recent research BID3 suggests that the discrepancy between the intrinsic and ambient dimensionalities of data also contributes to the difficulties in training GANs and VAEs.In this work, we present a novel framework for training autoencoder-based generative models, with non-adversarial losses and unrestricted neural network architectures. Given a standard autoencoder and a target data distribution, instead of matching the target distribution in the data space, we map both the generated and target distributions to the latent space using the encoder, and train the generator (or decoder) to minimize the divergence between the mapped distributions. We prove, under mild assumptions, that by minimizing a form of latent reconstruction error, matching the target distribution in the latent space implies matching it in the data space. We call this framework perceptual generative autoencoder (PGA). We show that PGA enables training generative autoencoders with maximum likelihood, without restrictions on architectures or latent dimensionalities. In addition, when combined with VAE, PGA can generate sharper samples than vanilla VAE. 1 2 METHODS 2.1 PERCEPTUAL GENERATIVE MODEL Let f : R D → R H be the encoder parameterized by φ, and g : R H → R D be the decoder parameterized by θ. Our goal is to obtain a generative model, which maps a simple prior distribution to the data distribution, D. Throughout this paper, we use N (0, I) as the prior distribution.For z ∈ R H , the output of the decoder, g (z), lies in a manifold that is at most H-dimensional. Therefore, if we train the autoencoder to minimize DISPLAYFORM0 DISPLAYFORM1 , thenx can be seen as a projection of the input data, x, onto the manifold of g (z). LetD denote the distribution ofx. Given enough capacity of the encoder,D is the best approximation to D (in terms of L2 distance), that we can obtain from the decoder, and thus can serve as a surrogate target for training the generator.Due to the difficulty of directly training the generator to matchD, we seek to mapD to the latent space, and train the generator to match the mapped distribution,Ĥ, in the latent space. To this end, we reuse the encoder for mappingD toĤ, and train the generator such that h (·) = f (g (·)) maps N (0, I) toĤ. In addition, to ensure that g maps N (0, I) toD, we minimize the following latent reconstruction loss with respect to (w.r.t.) φ: DISPLAYFORM2 Formally, let Z (x) be the set of all z's that are mapped to the same x by the decoder, we have the following theorem: Theorem 1. Assuming the convexity of Z (x) for all x ∈ R D , and sufficient capacity of the encoder; for z ∼ N (0, I), if Eq. (2) is minimized and h (z) ∼Ĥ, then g (z) ∼D.Proof. We first show that any different x's generated by the decoder are mapped to different z's by the encoder. Let x 1 = g (z 1 ), x 2 = g (z 2 ), and x 1 = x 2 . Since the encoder has sufficient capacity and Eq. (2) is minimized, we have f ( DISPLAYFORM3 For z ∼ N (0, I), denote the distributions of g (z) and h (z), respectively, by D and H. We then consider the case where D andD are discrete distributions. If g (z) D , then there exists an x, DISPLAYFORM4 The result still holds when D andD approach continuous distributions.Note that the two distributions compared in Theorem 1, D andD, are mapped respectively from N (0, I) and H. While N (0, I) is supported on the whole R H , there can be z's with low probabilities in N (0, I), but with high probabilities in H, which are not well covered by Eq. (2). Therefore, it is sometimes helpful to minimize another latent reconstruction loss on H: DISPLAYFORM5 By Theorem 1, the problem of training the generative model reduces to training h to map N (0, I) tô H, which we refer to as the perceptual generative model. In the subsequent subsections, we present a maximum likelihood approach, as well as a VAE-based approach, to train the perceptual generative model. We proposed a framework, PGA, for training autoencoder-based generative models, with nonadversarial losses and unrestricted neural network architectures. By matching target distributions in the latent space, PGA trained with maximum likelihood generalizes the idea of reversible generative models to unrestricted neural network architectures and arbitrary latent dimensionalities. In addition, it improves the performance of VAE when combined together.In principle, PGA can be combined with any method that can train the perceptual generative model. While we have only considered two non-adversarial approaches, an interesting future work would be to combine PGA with an adversarial discriminator trained on latent representations. Moreover, the compatibility issue with batch normalization deserves further investigation.
A framework for training autoencoder-based generative models, with non-adversarial losses and unrestricted neural network architectures. This paper uses autoencoders to do distribution matching in high dimensional space.