The importance-weighted autoencoder (IWAE) approach of Burda et al. defines a sequence of increasingly tighter bounds on the marginal likelihood of latent variable models. Recently, Cremer et al. reinterpreted the IWAE bounds as ordinary variational evidence lower bounds (ELBO) applied to increasingly accurate variational distributions. In this work, we provide yet another perspective on the IWAE bounds. We interpret each IWAE bound as a biased estimator of the true marginal likelihood where for the bound defined on $K$ samples we show the bias to be of order O(1/K). In our theoretical analysis of the IWAE objective we derive asymptotic bias and variance expressions. Based on this analysis we develop jackknife variational inference (JVI),
 a family of bias-reduced estimators reducing the bias to $O(K^{-(m+1)})$ for any given m < K while retaining computational efficiency. Finally, we demonstrate that JVI leads to improved evidence estimates in variational autoencoders. We also report first results on applying JVI to learning variational autoencoders.

 Our implementation is available at https://github.com/Microsoft/jackknife-variational-inference Variational autoencoders (VAE) are a class of expressive probabilistic deep learning models useful for generative modeling, representation learning, and probabilistic regression. Originally proposed in BID8 and BID22 , VAEs consist of a probabilistic model as well as an approximate method for maximum likelihood estimation. In the generative case, the model is defined as DISPLAYFORM0 where z is a latent variable, typically a high dimensional vector; the corresponding prior distribution p(z) is fixed and typically defined as a standard multivariate Normal distribution N (0, I). To achieve an expressive marginal distribution p(x), we define p θ (x|z) through a neural network, making the model (1) a deep probabilistic model.Maximum likelihood estimation of the parameters θ in (1) is intractable, but BID8 and BID22 propose to instead maximize the evidence lower-bound (ELBO), log p(x) ≥ E z∼qω(z|x) log p θ (x|z) p(z) q ω (z|x)=: L E .Here , q ω (z|x) is an auxiliary inference network, parametrized by ω. Simultaneous optimization of (2) over both θ and ω performs approximate maximum likelihood estimation in the model p(x) of FORMULA0 and forms the standard VAE estimation method. 1 The implementation is available at https://github.com/Microsoft/ jackknife-variational-inferenceIn practice L E is estimated using Monte Carlo: we draw K samples z i ∼ q ω (z|x), then use the unbiased estimatorL E of L E ,L DISPLAYFORM1 The VAE approach is empirically very successful but are there fundamental limitations? One limitation is the quality of the model p θ (x|z): this model needs to be expressive enough to model the true distribution over x. Another limitation is that L E is only a lower-bound to the true likelihood. Is this bound tight? It can be shown, BID8 , that when q(z|x) = p(z|x) we have L E = log p(x), hence (2) becomes exact. Therefore, we should attempt to choose an expressive class of distributions q(z|x) and indeed recent work has extensively investigated richer variational families. We discuss these methods in Section 7 but now review the importance weighted autoencoder (IWAE) method we build upon. In summary we proposed to leverage classic higher-order bias removal schemes for evidence estimation. Our approach is simple to implement, computationally efficient, and clearly improves over existing evidence approximations based on variational inference. More generally our jackknife variational inference debiasing formula can also be used to debias log-evidence estimates coming from annealed importance sampling.However, one surprising finding from our work is that using our debiased estimates for training VAE models did not improve over the IWAE training objective and this is surprising because apriori a better evidence estimate should allow for improved model learning.One possible extension to our work is to study the use of other resampling methods for bias reduction; promising candidates are the iterated bootstrap, the Bayesian bootstrap, and the debiasing lemma. These methods could offer further improvements on bias reduction or reduced variance, however, the key challenge is to overcome computational requirements of these methods or, alternatively, to derive key quantities analytically. 6 Application of the debiasing lemma in particular requires the careful construction of a truncation distribution and often produces estimators of high variance.While variance reduction plays a key role in certain areas of machine learning, our hope is that our work shows that bias reduction techniques are also widely applicable. DISPLAYFORM0 Note that only Y K is random in (28), all other quantities are constant. Therefore, by taking the expectation on the left and right side of FORMULA1 we obtain DISPLAYFORM1 The right hand side of FORMULA1 is expressed in terms of the central moments for i ≥ 2, DISPLAYFORM2 of Y K , whereas we are interested in an expression using the central moments i ≥ 2, DISPLAYFORM3 we denote the shared first non-central moment. Because Y K is a sample mean we can use existing results that relate γ i to µ i . In particular (Angelova, 2012, Theorem 1) gives the relations DISPLAYFORM4 DISPLAYFORM5 Expanding FORMULA1 to order five and using the relations FORMULA2 to FORMULA2 gives DISPLAYFORM6 Regrouping the terms by order of K produces the result (8).
Variational inference is biased, let's debias it. Introduces jackknife variational inference, a method for debiasing Monte Carlo objectives such as the importance weighted auto-encoder. The authors analyze the bias and variance of the IWAE bound and derive a jacknife approach to estimate moments as a way to debias IWAE for finite importance weighted samples.