Building on the success of deep learning, two modern approaches to learn a probability model of the observed data are Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs). VAEs consider an explicit probability model for the data and compute a generative distribution by maximizing a variational lower-bound on the log-likelihood function. GANs, however, compute a generative model by minimizing a distance between observed and generated probability distributions without considering an explicit model for the observed data. The lack of having explicit probability models in GANs prohibits computation of sample likelihoods in their frameworks and limits their use in statistical inference problems. In this work, we show that an optimal transport GAN with the entropy regularization can be viewed as a generative model that maximizes a lower-bound on average sample likelihoods, an approach that VAEs are based on. In particular, our proof constructs an explicit probability model for GANs that can be used to compute likelihood statistics within GAN's framework. Our numerical results on several datasets demonstrate consistent trends with the proposed theory. Learning generative models is becoming an increasingly important problem in machine learning and statistics with a wide range of applications in self-driving cars BID26 , robotics BID10 , natural language processing BID14 , domain-transfer BID25 , computational biology BID6 , etc. Two modern approaches to deal with this problem are Generative Adversarial Networks (GANs) BID7 and Variational AutoEncoders (VAEs) BID13 BID15 BID23 BID28 BID17 .VAEs BID13 ) compute a generative model by maximizing a variational lowerbound on average sample likelihoods using an explicit probability distribution for the data. GANs , however, learn a generative model by minimizing a distance between observed and generated distributions without considering an explicit probability model for the data. Empirically , GANs have been shown to produce higher-quality generative samples than that of VAEs BID12 . However, since GANs do not consider an explicit probability model for the data, we are unable to compute sample likelihoods using their generative models. Computations of sample likelihoods and posterior distributions of latent variables are critical in several statistical inference. Inability to obtain such statistics within GAN's framework severely limits their applications in such statistical inference problems.In this paper, we resolve these issues for a general formulation of GANs by providing a theoreticallyjustified approach to compute sample likelihoods using GAN's generative model. Our results can open new directions to use GANs in massive-data applications such as model selection, sample selection, hypothesis-testing, etc (see more details in Section 5). Now, we state our main results informally without going into technical conditions while precise statements of our results are presented in Section 2. Let Y andŶ ∶= G(X) represent observed (i.e. real) and generative (i.e. fake or synthetic) variables, respectively. X (i.e. the latent variable) is the randomness used as the input to the generator G(.). Consider the following explicit probability By training a GAN model, we first compute optimal generator G * and optimal coupling between the observed variable Y and the latent variable X. The likelihood of a test sample y test can then be lower-bounded using a combination of three terms: (1) the expected distance of y test to the distribution learnt by the generative model, (2) the entropy of the coupled latent variable given y test and (3) the likelihood of the coupled latent variable with y test .model of the data given a latent sample X = x: f Y X=x (y) ∝ exp(− (y, G(x))), (1.1)where (., .) is a loss function. f Y X=x (y) is the model that we are considering for the underlying data distribution. This is a reasonable model for the data as the function G can be a complex function. Similar data models have been used in VAEs . Under this explicit probability model, we show that minimizing the objective of an optimal transport GAN (e.g. Wasserstein BID0 ) with the cost function (., .) and an entropy regularization BID2 BID27 ) maximizes a variational lower-bound on average sample likelihoods. I.e.average sample likelihoods ≥ − (entropic GAN objective) + constants.(1.2)If (y,ŷ) = y −ŷ 2 , the optimal transport (OT) GAN simplifies to WGAN while if (y,ŷ) = y −ŷ 2 2 , the OT GAN simplifies to the quadratic GAN (or, W2GAN) BID3 ). The precise statement of this result can be found in Theorem 1. This result provides a statistical justification for GAN's optimization and puts it in par with VAEs whose goal is to maximize a lower bound on sample likelihoods. We note that the entropy regularization has been proposed primarily to improve computational aspects of GANs BID5 . Our results provide an additional statistical justification for this regularization term. Moreover, using GAN's training, we obtain a coupling between the observed variable Y and the latent variable X. This coupling provides the conditional distribution of the latent variable X given an observed sample Y = y. The explicit model of equation 1.1 acts similar to the decoder in the VAE framework, while the coupling computed using GANs acts as an encoder.Connections between GANs and VAEs have been investigated in some of the recent works as well BID11 BID16 . In BID11 , GANs are interpreted as methods performing variational inference on a generative model in the label space. In their framework, observed data samples are treated as latent variables while the generative variable is the indicator of whether data is real or fake. The method in BID16 , on the other hand, uses an auxiliary discriminator network to rephrase the maximum-likelihood objective of a VAE as a twoplayer game similar to the objective of a GAN. Our method is different from both these approaches as we consider an explicit probability model for the data, and show that the entropic GAN objective maximizes a variational lower bound under this probability model, thus allowing sample likelihood computation in GANs similar to VAEs.Of relevance to our work is BID30 , in which annealed importance sampling (AIS) is used to evaluate the approximate likelihood of decoder-based generative models. More specifically, a Gaussian observation model with a fixed variance is used as the generative distribution for GANbased models on which the AIS is computed. Gaussian observation models may not be proper specially in high-dimensional spaces . Our approach, on the other hand, makes a connection between GANs and VAEs by constructing a theoretically-motivated model for the data distribution in GANs. We then leverage this approach in computing sample likelihood estimates in GANs.Another key question that we address here is how to estimate the likelihood of a new sample y test given the generative model trained using GANs. For instance, if we train a GAN on stop-sign images, upon receiving a new image, one may wish to compute the likelihood of the new sample y test according to the trained generative model. In standard GAN formulations, the support of the generative distribution lies on the range of the optimal generator function. Thus, if the observed sample y test does not lie on that range (which is very likely in practice ), there is no way to assign a sensible likelihood score to that sample. Below, we show that using the explicit probability model of equation 1.1, we can lower-bound the likelihood of this sample y test . This is similar to the variational lower-bound on sample likelihoods used in VAEs. Our numerical results show that this lower-bound well-reflect the expected trends of the true sample likelihoods.Let G * and P * Y,X be the optimal generator and the optimal coupling between real and latent variables, respectively. The optimal coupling P * Y,X can be computed efficiently for entropic GANs as we explain in Section 3. For other GAN architectures, one may approximate such couplings as we explain in Section 4. The log likelihood of a new test sample y test can be lower-bounded as DISPLAYFORM0 distance to the generative model DISPLAYFORM1 We present the precise statement of this result in Corollary 2. This result combines three components in order to approximate the likelihood of a sample given a trained generative model:• The distance between y test to the generative model. If this distance is large, the likelihood of observing y test from the generative model is small.• The entropy of the coupled latent variable. If the entropy term is large, the coupled latent variable has a large randomness. This contributes positively to the sample likelihood.• The likelihood of the coupled latent variable. If latent samples have large likelihoods, the likelihood of the observed test sample will be large as well. FIG2 provides a pictorial illustration of these components. In what follows, we explain the technical ingredients of our main results. In Section 3, we present computational methods for GANs and entropic GANs, while in Section 4, we provide numerical experiments on benchmark datasets. In this paper, we have provided a statistical framework for a family of GANs. Our main result shows that the entropic GAN optimization can be viewed as maximization of a variational lower-bound on average log-likelihoods, an approach that VAEs are based upon. This result makes a connection between two most-popular generative models, namely GANs and VAEs. More importantly, our result constructs an explicit probability model for GANs that can be used to compute a lower-bound on sample likelihoods. Our experimental results on various datasets demonstrate that this likelihood surrogate can be a good approximation of the true likelihood function. Although in this paper we mainly focus on understanding the behavior of the sample likelihood surrogate in different datasets, the proposed statistical framework of GANs can be used in various statistical inference applications. For example, our proposed likelihood surrogate can be used as a quantitative measure to evaluate the performance of different GAN architectures, it can be used to quantify the domain shifts, it can be used to select a proper generator class by balancing the bias term vs. variance, it can be used to detect outlier samples, it can be used in statistical tests such as hypothesis testing, etc. We leave exploring these directions for future work. APPENDIX A PROOF OF THEOREM 1Using the Baye's rule, one can compute the log-likelihood of an observed sample y as follows: DISPLAYFORM0 where the second step follows from equation 2.4.Consider a joint density function P X,Y such that its marginal distributions match P X and P Y . Note that the equation A.1 is true for every x. Thus, we can take the expectation of both sides with respect to a distribution P X Y =y . This leads to the following equation: DISPLAYFORM1 where H(.) is the Shannon-entropy function.Next we take the expectation of both sides with respect to P Y : DISPLAYFORM2 Here, we replaced the expectation over P X with the expectation over f X since one can generate an arbitrarily large number of samples from the generator. Since the KL divergence is always nonnegative, we have DISPLAYFORM3 Moreover, using the data processing inequality, we have BID1 . Thus, DISPLAYFORM4 DISPLAYFORM5 GAN objective with entropy regularizer DISPLAYFORM6 This inequality is true for every P X,Y satisfying the marginal conditions. Thus, similar to VAEs, we can pick P X,Y to maximize the lower bound on average sample log-likelihoods. This leads to the entropic GAN optimization 2.3.
A statistical approach to compute sample likelihoods in Generative Adversarial Networks Show that WGAN with entropic regularization maximizes a lower bound on the likelihood of the observed data distribution. Authors claim it is possible to leverage the upper bound from an entropy regularized optimal transport to come up with a measure of 'sample likelihood'.