Deep generative models have achieved impressive success in recent years. Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), as powerful frameworks for deep generative model learning, have largely been considered as two distinct paradigms and received extensive independent studies respectively. This paper aims to establish formal connections between GANs and VAEs through a new formulation of them. We interpret sample generation in GANs as performing posterior inference, and show that GANs and VAEs involve minimizing KL divergences of respective posterior and inference distributions with opposite directions, extending the two learning phases of classic wake-sleep algorithm, respectively. The unified view provides a powerful tool to analyze a diverse set of existing model variants, and enables to transfer techniques across research lines in a principled way. For example, we apply the importance weighting method in VAE literatures for improved GAN learning, and enhance VAEs with an adversarial mechanism that leverages generated samples. Experiments show generality and effectiveness of the transfered techniques. Deep generative models define distributions over a set of variables organized in multiple layers. Early forms of such models dated back to works on hierarchical Bayesian models BID30 and neural network models such as Helmholtz machines , originally studied in the context of unsupervised learning, latent space modeling, etc. Such models are usually trained via an EM style framework, using either a variational inference (Jordan et al., 1999) or a data augmentation BID35 algorithm. Of particular relevance to this paper is the classic wake-sleep algorithm dates by for training Helmholtz machines, as it explored an idea of minimizing a pair of KL divergences in opposite directions of the posterior and its approximation.In recent years there has been a resurgence of interests in deep generative modeling. The emerging approaches, including Variational Autoencoders (VAEs) BID18 , Generative Adversarial Networks (GANs) BID12 , Generative Moment Matching Networks (GMMNs) BID24 BID10 , auto-regressive neural networks BID21 BID37 , and so forth, have led to impressive results in a myriad of applications, such as image and text generation (Radford et al., 2015; Hu et al., 2017; BID37 , disentangled representation learning BID7 BID20 , and semi-supervised learning (Salimans et al., 2016; BID19 . The deep generative model literature has largely viewed these approaches as distinct model training paradigms. For instance, GANs aim to achieve an equilibrium between a generator and a discriminator; while VAEs are devoted to maximizing a variational lower bound of the data log-likelihood. A rich array of theoretical analyses and model extensions have been developed independently for GANs BID0 BID1 Salimans et al., 2016; BID31 and VAEs BID4 BID8 Hu et al., 2017) , respectively. A few works attempt to combine the two objectives in a single model for improved inference and sample generation BID26 BID22 BID25 BID34 . Despite the significant progress specific to each method, it remains unclear how these apparently divergent approaches connect to each other in a principled way.In this paper, we present a new formulation of GANs and VAEs that connects them under a unified view, and links them back to the classic wake-sleep algorithm. We show that GANs and VAEs involve minimizing opposite KL divergences of respective posterior and inference distributions, and extending the sleep and wake phases, respectively, for generative model learning. More specifically, we develop a reformulation of GANs that interprets generation of samples as performing posterior inference, leading to an objective that resembles variational inference as in VAEs. As a counterpart, VAEs in our interpretation contain a degenerated adversarial mechanism that blocks out generated samples and only allows real examples for model training.The proposed interpretation provides a useful tool to analyze the broad class of recent GAN-and VAEbased algorithms, enabling perhaps a more principled and unified view of the landscape of generative modeling. For instance, one can easily extend our formulation to subsume InfoGAN BID7 that additionally infers hidden representations of examples, VAE/GAN joint models BID22 BID5 ) that offer improved generation and reduced mode missing, and adversarial domain adaptation (ADA) BID11 Purushotham et al., 2017) that is traditionally framed in the discriminative setting.The close parallelisms between GANs and VAEs further ease transferring techniques that were originally developed for improving each individual class of models, to in turn benefit the other class. We provide two examples in such spirit: 1) Drawn inspiration from importance weighted VAE (IWAE) BID4 , we straightforwardly derive importance weighted GAN (IWGAN) that maximizes a tighter lower bound on the marginal likelihood compared to the vanilla GAN. 2) Motivated by the GAN adversarial game we activate the originally degenerated discriminator in VAEs, resulting in a full-fledged model that adaptively leverages both real and fake examples for learning. Empirical results show that the techniques imported from the other class are generally applicable to the base model and its variants, yielding consistently better performance. Our new interpretations of GANs and VAEs have revealed strong connections between them, and linked the emerging new approaches to the classic wake-sleep algorithm. The generality of the proposed formulation offers a unified statistical insight of the broad landscape of deep generative modeling, and encourages mutual exchange of techniques across research lines. One of the key ideas in our formulation is to interpret sample generation in GANs as performing posterior inference. This section provides a more general discussion of this point.Traditional modeling approaches usually distinguish between latent and visible variables clearly and treat them in very different ways. One of the key thoughts in our formulation is that it is not necessary to make clear boundary between the two types of variables (and between generation and inference), but instead, treating them as a symmetric pair helps with modeling and understanding. For instance, we treat the generation space x in GANs as latent, which immediately reveals the connection between GANs and adversarial domain adaptation, and provides a variational inference interpretation of the generation. A second example is the classic wake-sleep algorithm, where the wake phase reconstructs visibles conditioned on latents, while the sleep phase reconstructs latents conditioned on visibles (i.e., generated samples). Hence, visible and latent variables are treated in a completely symmetric manner.• Empirical data distributions are usually implicit, i.e., easy to sample from but intractable for evaluating likelihood. In contrast, priors are usually defined as explicit distributions, amiable for likelihood evaluation.• The complexity of the two distributions are different. Visible space is usually complex while latent space tends (or is designed) to be simpler.However, the adversarial approach in GANs and other techniques such as density ratio estimation (Mohamed & Lakshminarayanan, 2016) and approximate Bayesian computation BID2 have provided useful tools to bridge the gap in the first point. For instance , implicit generative models such as GANs require only simulation of the generative process without explicit likelihood evaluation, hence the prior distributions over latent variables are used in the same way as the empirical data distributions, namely, generating samples from the distributions. For explicit likelihood-based models, adversarial autoencoder (AAE) leverages the adversarial approach to allow implicit prior distributions over latent space. Besides, a few most recent work BID26 BID36 BID34 Rosca et al., 2017) extends VAEs by using implicit variational distributions as the inference model. Indeed, the reparameterization trick in VAEs already resembles construction of implicit variational distributions (as also seen in the derivations of IWGANs in Eq.37). In these algorithms, adversarial approach is used to replace intractable minimization of the KL divergence between implicit variational distributions and priors.The second difference in terms of space complexity guides us to choose appropriate tools (e.g., adversarial approach v.s. reconstruction optimization, etc) to minimize the distance between distributions to learn and their targets. However, the tools chosen do not affect the underlying modeling mechanism.For instance, VAEs and adversarial autoencoder both regularize the model by minimizing the distance between the variational posterior and certain prior, though VAEs choose KL divergence loss while AAE selects adversarial loss.We can further extend the symmetric treatment of visible/latent x/z pair to data/label x/t pair, leading to a unified view of the generative and discriminative paradigms for unsupervised and semi-supervised learning. Specifically, conditional generative models create (data, label) pairs by generating data x given label t. These pairs can be used for classifier training (Hu et al., 2017; BID32 . In parallel, discriminative approaches such as knowledge distillation BID14 BID17 create (data, label) pairs by generating label t conditioned on data x. With the symmetric view of x and t spaces , and neural network based black-box mappings across spaces, we can see the two approaches are essentially the same. A ADVERSARIAL DOMAIN ADAPTATION (ADA)ADA aims to transfer prediction knowledge learned from a source domain with labeled data to a target domain without labels, by learning domain-invariant features. Let D φ (x) = q φ (y|x) be the domain discriminator . The conventional formulation of ADA is as following : DISPLAYFORM0 Further add the supervision objective of predicting label t(z) of data z in the source domain, with a classifier f ω (t|x) parameterized with π: DISPLAYFORM1 We then obtain the conventional formulation of adversarial domain adaptation used or similar in BID11 Purushotham et al., 2017) .B PROOF OF LEMMA 1Proof. DISPLAYFORM2 where DISPLAYFORM3 Note that p θ (x|y = 0) = p g θ (x), and p θ (x|y = 1) = p data (x). DISPLAYFORM4 . Eq.(21) can be simplified as: DISPLAYFORM5 On the other hand , DISPLAYFORM6 Note that DISPLAYFORM7 Taking derivatives of Eq. FORMULA1 w.r.t θ at θ 0 we get DISPLAYFORM8 Taking derivatives of the both sides of Eq. FORMULA1 at w.r.t θ at θ 0 and plugging the last equation of Eq. FORMULA1 , we obtain the desired results. We show that, in Lemma.1 (Eq.6), the JSD term is upper bounded by the KL term, i.e., DISPLAYFORM9 DISPLAYFORM10 Proof. From Eq. FORMULA1 , we have DISPLAYFORM11 From Eq. FORMULA1 and Eq. FORMULA1 , we have DISPLAYFORM12 Eq. FORMULA1 and Eq. FORMULA1
A unified statistical view of the broad class of deep generative models  The paper develops a framework interpreting GAN algorithms as performing a form of variational inference on a generative model reconstructing an indicator variable of whether a sample is from the true of generative data distributions.