Although variational autoencoders (VAEs) represent a widely influential deep generative model, many aspects of the underlying energy function remain poorly understood.  In particular, it is commonly believed that Gaussian encoder/decoder assumptions reduce the effectiveness of VAEs in generating realistic samples .  In this regard, we rigorously analyze the VAE objective, differentiating situations where this belief is and is not actually true .  We then leverage the corresponding insights to develop a simple VAE enhancement that requires no additional hyperparameters or sensitive tuning .  Quantitatively, this proposal produces crisp samples and stable FID scores that are actually competitive with a variety of GAN models, all while retaining desirable attributes of the original VAE architecture . The code for our model is available at \url{https://github.com/daib13/TwoStageVAE}. Our starting point is the desire to learn a probabilistic generative model of observable variables x ∈ χ, where χ is a r-dimensional manifold embedded in R d . Note that if r = d, then this assumption places no restriction on the distribution of x ∈ R d whatsoever; however, the added formalism is introduced to handle the frequently encountered case where x possesses low-dimensional structure relative to a high-dimensional ambient space, i.e., r d. In fact, the very utility of generative models of continuous data, and their attendant low-dimensional representations, often hinges on this assumption BID1 . It therefore behooves us to explicitly account for this situation.Beyond this, we assume that χ is a simple Riemannian manifold, which means there exists a diffeomorphism ϕ between χ and R r , or more explicitly, the mapping ϕ : χ → R r is invertible and differentiable. Denote a ground-truth probability measure on χ as µ gt such that the probability mass of an infinitesimal dx on the manifold is µ gt (dx) and χ µ gt (dx) = 1.The variational autoencoder (VAE) BID17 BID28 attempts to approximate this ground-truth measure using a parameterized density p θ (x) defined across all of R d since any underlying generative manifold is unknown in advance. This density is further assumed to admit the latent decomposition p θ (x) = p θ (x|z)p(z)dz, where z ∈ R κ serves as a lowdimensional representation, with κ ≈ r and prior p(z) = N (z|0, I).Ideally we might like to minimize the negative log-likelihood − log p θ (x) averaged across the ground-truth measure µ gt , i.e., solve min θ χ − log p θ (x)µ gt (dx). Unfortunately though, the required marginalization over z is generally infeasible. Instead the VAE model relies on tractable encoder q φ (z|x) and decoder p θ (x|z) distributions, where φ represents additional trainable parameters. The canonical VAE cost is a bound on the average negative log-likelihood given by L(θ, φ) χ {− log p θ (x) + KL [q φ (z|x)||p θ (z|x)]} µ gt (dx) ≥ χ − log p θ (x)µ gt (dx),where the inequality follows directly from the non-negativity of the KL-divergence. Here φ can be viewed as tuning the tightness of bound, while θ dictates the actual estimation of µ gt . Using a few standard manipulations, this bound can also be expressed as DISPLAYFORM0 which explicitly involves the encoder/decoder distributions and is conveniently amenable to SGD optimization of {θ, φ} via a reparameterization trick BID17 BID28 . The first term in (2 ) can be viewed as a reconstruction cost (or a stochastic analog of a traditional autoencoder), while the second penalizes posterior deviations from the prior p(z). Additionally, for any realizable implementation via SGD, the integration over χ must be approximated via a finite sum across training samples {x (i) } n i=1 drawn from µ gt . Nonetheless, examining the true objective L(θ, φ) can lead to important, practically-relevant insights.At least in principle, q φ (z|x) and p θ (x|z) can be arbitrary distributions, in which case we could simply enforce q φ (z|x) = p θ (z|x) ∝ p θ (x|z)p(z) such that the bound from (1) is tight. Unfortunately though, this is essentially always an intractable undertaking. Consequently, largely to facilitate practical implementation, a commonly adopted distributional assumption for continuous data is that both q φ (z|x) and p θ (x|z) are Gaussian. This design choice has previously been cited as a key limitation of VAEs BID5 BID18 , and existing quantitative tests of generative modeling quality thus far dramatically favor contemporary alternatives such as generative adversarial networks (GAN) BID13 . Regardless, because the VAE possesses certain desirable properties relative to GAN models (e.g., stable training BID29 , interpretable encoder/inference network BID4 , outlier-robustness BID9 , etc.), it remains a highly influential paradigm worthy of examination and enhancement.In Section 2 we closely investigate the implications of VAE Gaussian assumptions leading to a number of interesting diagnostic conclusions. In particular, we differentiate the situation where r = d, in which case we prove that recovering the ground-truth distribution is actually possible iff the VAE global optimum is reached, and r < d, in which case the VAE global optimum can be reached by solutions that reflect the ground-truth distribution almost everywhere, but not necessarily uniquely so. In other words, there could exist alternative solutions that both reach the global optimum and yet do not assign the same probability measure as µ gt .Section 3 then further probes this non-uniqueness issue by inspecting necessary conditions of global optima when r < d. This analysis reveals that an optimal VAE parameterization will provide an encoder/decoder pair capable of perfectly reconstructing all x ∈ χ using any z drawn from q φ (z|x). Moreover, we demonstrate that the VAE accomplishes this using a degenerate latent code whereby only r dimensions are effectively active. Collectively, these results indicate that the VAE global optimum can in fact uniquely learn a mapping to the correct ground-truth manifold when r < d, but not necessarily the correct probability measure within this manifold, a critical distinction.Next we leverage these analytical results in Section 4 to motivate an almost trivially-simple, twostage VAE enhancement for addressing typical regimes when r < d. In brief, the first stage just learns the manifold per the allowances from Section 3, and in doing so, provides a mapping to a lower dimensional intermediate representation with no degenerate dimensions that mirrors the r = d regime. The second (much smaller) stage then only needs to learn the correct probability measure on this intermediate representation, which is possible per the analysis from Section 2. Experiments from Sections 5 and 6 empirically corroborate motivational theory and reveal that the proposed two-stage procedure can generate high-quality samples, reducing the blurriness often attributed to VAE models in the past BID11 BID21 . And to the best of our knowledge, this is the first demonstration of a VAE pipeline that can produce stable FID scores, an influential recent metric for evaluating generated sample quality BID16 , that are comparable to GAN models under neutral testing conditions. Moreover, this is accomplished without additional penalties, cost function modifications, or sensitive tuning parameters. Finally, an extended version of this work can be found in BID8 ). There we include additional results, consideration of disentangled representations, as well as a comparative discussion of broader VAE modeling paradigms such as those involving normalizing flows or parameterized families for p(z). It is often assumed that there exists an unavoidable trade-off between the stable training, valuable attendant encoder network, and resistance to mode collapse of VAEs, versus the impressive visual quality of images produced by GANs. While we certainly are not claiming that our two-stage VAE model is superior to the latest and greatest GAN-based architecture in terms of the realism of generated samples, we do strongly believe that this work at least narrows that gap substantially such that VAEs are worth considering in a broader range of applications. For further results and discussion, including consideration of broader VAE modeling paradigms and the identifiability of disentangled representations, please see BID8 .
We closely analyze the VAE objective function and draw novel conclusions that lead to simple enhancements. Proposes a two-stage VAE method to generate high-quality samples and avoid blurriness. This paper analyzes the Gaussian VAEs. The paper provides a number of theoretical results on "vanilla" Gaussian Variational Auto-Encoders, which are then used to build a new algorithm called "2 stage VAEs".