Due to the phenomenon of “posterior collapse,” current latent variable generative models pose a challenging design choice that either weakens the capacity of the decoder or requires altering the training objective. We develop an alternative that utilizes the most powerful generative models as decoders, optimize the variational lower bound, and ensures that the latent variables preserve and encode useful information. Our proposed δ-VAEs achieve this by constraining the variational family for the posterior to have a minimum distance to the prior. For sequential latent variable models, our approach resembles the classic representation learning approach of slow feature analysis. We demonstrate our method’s efficacy at modeling text on LM1B and modeling images: learning representations, improving sample quality, and achieving state of the art log-likelihood on CIFAR-10 and ImageNet 32 × 32. Deep latent variable models trained with amortized variational inference have led to advances in representation learning on high-dimensional datasets BID25 BID33 . These latent variable models typically have simple decoders, where the mapping from the latent variable to the input space is unimodal, for example using a conditional Gaussian decoder. This typically results in representations that are good at capturing the global structure in the input, but fail at capturing more complex local structure (e.g. texture BID28 ). In parallel, advances in autoregressive models have led to drastic improvements in density modeling and sample quality without explicit latent variables BID39 . While these models are good at capturing local statistics, they often fail to produce globally coherent structures BID30 .Combining the power of tractable densities from autoregressive models with the representation learning capabilities of latent variable models could result in higher-quality generative models with useful latent representations. While much prior work has attempted to join these two models, a common problem remains. If the autoregressive decoder is expressive enough to model the data density, then the model can learn to ignore the latent variables, resulting in a trivial posterior that collapses to the prior. This phenomenon has been frequently observed in prior work and has been referred to as optimization challenges of VAEs by BID5 , the information preference property by , and the posterior collapse problems by several others (e.g. van den BID40 BID14 . Ideally, an approach that mitigates posterior collapse would not alter the evidence lower bound (ELBO) training objective, and would allow the practitioner to leverage the most recent advances in powerful autoregressive decoders to improve performance. To the best of our knowledge , no prior work has succeeded at this goal. Most common approaches either change the objective BID20 BID1 Zhao et al., 2017; BID29 Goyal et al., 2017) , or weaken the decoder BID5 BID18 . Additionally, these approaches are often challenging to tune and highly sensitive to hyperparameters BID1 .In this paper, we propose δ-VAEs , a simple framework for selecting variational families that prevent posterior collapse without altering the ELBO training objective or weakening the decoder. By restricting the parameters or family of the posterior, we ensure that there is a minimum KL divergence, δ, between the posterior and the prior.We demonstrate the effectiveness of this approach at learning latent-variable models with powerful decoders on images (CIFAR-10, and ImageNet 32 × 32), and text (LM1B). We achieve state of the art log-likelihood results with image models by additionally introducing a sequential latent-variable model with an anti-causal encoder structure. Our experiments demonstrate the utility of δ-VAEs at learning useful representations for downstream tasks without sacrificing performance on density modeling. In this work, we have demonstrated that δ-VAEs provide a simple, intuitive, and effective solution to posterior collapse in latent variable models, enabling them to be paired with powerful decoders. Unlike prior work, we do not require changes to the objective or weakening of the decoder, and we can learn useful representations as well as achieving state-of-the-art likelihoods. While our work presents two simple posterior-prior pairs, there are a number of other possibilities that could be explored in future work. Our work also points to at least two interesting challenges for latentvariable models: (1) can they exceed the performance of a strong autoregressive baseline, and (2) can they learn representations that improve downstream applications such as classification? DISPLAYFORM0 B DERIVATION OF THE KL-DIVERGENCE BETWEEN AR(1) AND DIAGONAL GAUSSIAN, AND ITS LOWER-BOUND DISPLAYFORM1 Noting the analytic form for the KL-divergence for two uni-variate Gaussian distributions: DISPLAYFORM2 we now derive the lower-bound for KL-divergence. To avoid clutter we assume a single dimension per timestep but extend the results to the general multivariate case at the end of this section. DISPLAYFORM3 C DERIVATION OF THE LOWER-BOUND Removing non-negative quadratic terms involving µ i in equation 3 and expanding back f inside the summation yields DISPLAYFORM4 Consider f a (x) = ax − ln(x) − 1 and its first and second order derivatives, f a (x) = a − 1 x and f a (x) ≥ 0. Thus, f a is convex and obtains its minimum value of ln(a) at x = a −1 . Substituting σ DISPLAYFORM5 When using multi-dimensional z i at each timestep, the committed rate is the sum of the KL for each individual dimension: DISPLAYFORM6 The most common choice for variational families is to assume that the components of the posterior are independent, for example using a multivariate Gaussian with a diagonal covariance: q φ (z|x) = N (z; µ q (x), σ q (x)). When paired with a standard Gaussian prior, p(z) = N (z; 0, 1), we can guarantee a committed information rate δ by constraining the mean and variance of the variational family (see Appendix C) DISPLAYFORM7 We can, thus, numerically solve DISPLAYFORM8 where the above equation has a solution for µ q , and the committed rate δ. Posterior parameters can thus be parameterised as: DISPLAYFORM9 Where φ parameterises the data-dependent part of µ q ad σ q , which allow the rate to go above the designated lower-bound δ.We compare this model with the temporal version of δ-VAE discussed in the paper and report the results in Table 3 . While independent δ-VAE also prevents the posterior from collapsing to prior, its performance in density modeling lags behind temporal δ-VAE.
 Avoid posterior collapse by lower bounding the rate. Presents an approach to preventing posterior collapse in VAEs by limiting the family of the variational approximation to the posterior This paper introduces a constraint on the family of variational posteriors such that the KL term can be controlled to combat posterior collapse in deep generative models such as VAEs