Powerful generative models, particularly in Natural Language Modelling, are commonly trained by maximizing a variational lower bound on the data log likelihood. These models often suffer from poor use of their latent variable, with ad-hoc annealing factors used to encourage retention of information in the latent variable. We discuss an alternative and general approach to latent variable modelling, based on an objective that encourages a perfect reconstruction by tying a stochastic autoencoder with a variational autoencoder (VAE). This ensures by design that the latent variable captures information about the observations, whilst retaining the ability to generate well. Interestingly, although our model is fundamentally different to a VAE, the lower bound attained is identical to the standard VAE bound but with the addition of a simple pre-factor; thus, providing a formal interpretation of the commonly used, ad-hoc pre-factors in training VAEs. Generative latent variable models are probabilistic models of observed data x of the form p(x, z) = p(x|z)p(z), where z is the latent variable. These models are widespread in machine learning and statistics. They are useful both because of their ability to generate new data and because the posterior p(z|x) provides insight into the low dimensional representation z corresponding to the high dimensional observation x. These latent z values are then often used in downstream tasks, such as topic modelling BID2 , multi-modal language modeling BID8 , and image captioning BID9 BID10 .Latent variable models, particularly in the form of Variational Autoencoders (VAEs) BID7 BID11 , have been successfully employed in natural language modelling tasks using varied architectures for both the encoder and the decoder BID0 BID2 BID12 BID16 BID13 . However , an architecture that is able to effectively capture meaningful semantic information into its latent variables is yet to be discovered.A VAE approach to language modelling was given by BID0 , the graphical model for which is shown in FIG0 (a). This forms a generative model p(x|z)p(z) of sentence x, based on latent variable z. Since the integral p(x) = p(x|z)p(z)dz is typically intractable, a common approach is to maximize the Evidence Lower Bound (ELBO) on the log likelihood, log p(x) ≥ log p(x|z) q(z|x) − D KL [q(z|x)||p(z)]where · q(z|x) is the expectation with respect to the variational distribution q(z|x), and D KL [·||·] represents the Kullback-Leibler (KL) divergence. Summing over all datapoints x gives a lower bound on the likelihood of the full dataset.In language modelling, typically both the generative model (decoder) p(x|z), and variational distribution (encoder) q(z|x), are parameterised using an LSTM recurrent neural networksee for example BID0 . This autoregressive generative model is so powerful that the maximum ELBO is achieved without making appreciable use of the latent variable in the model. Indeed, if trained using the SGVB algorithm BID7 , the model learns to ignore the latent representation and effectively relies solely on the decoder to generate good sentences. This is evidenced by the KL term in the objective function converging to zero, indicating that the approximate posterior distribution of the latent variable is trivially converging to its prior distribution.The dependency between what is represented by latent variables, and the capacity of the decoding distribution (i.e., its ability to model the data without using the latent) is a general phenomenon. BID16 used a lower capacity dilated CNN decoder to generate sentences, preventing the KL term going to zero. BID3 ; BID4 have discussed this in the context of image processing. A clear explanation of this phenomenon in terms of Bit-Back Coding is given in BID1 .A mechanism to avoid the model ignoring the latent entirely, while allowing a high capacity decoder is discussed in BID0 and uses an alternative training procedure called "KL annealing" -slowly turning on the KL term in the ELBO during training. KL annealing allows the model to use its latent variable to some degree by forcing the model into a local maximum of its objective function. Modifying the training procedure in this way to preferentially obtain local maxima suggests that the objective function used in BID0 may not be ideal for modelling language in such a way as to create a model that leverages its latent variables. We have seen that AutoGen successfully improves the fidelity of reconstructions from the latent variable as compared to VAEs. It does so in a principled way, by explicitly modelling both generation of the data and high-fidelity reconstruction. This is especially useful when the generative model is powerful, such as the autoregressive LSTM in BID0 .Other work toward enabling latent variables in VAE models to learn meaningful representations has focused on managing the structure of the representation, such as ensuring disentanglement. A detailed discussion of disentanglement in the context of VAEs is given by BID4 and its references. An example of disentangling representations in the context of image generation is BID3 , where the authors restrict the decoding model to describe only local information in the image (e.g., texture, shading), allowing their latents to describe global information (e.g., object geometry, overall color).Demanding high-fidelity reconstructions from latent variables in a model (e.g., AutoGen) is in tension with demanding specific information to be stored in the latent variables (e.g., disentanglement). This can be seen very clearly by comparing our work to BID4 , where the authors introduce an ad-hoc factor of β in front of the KL-divergence term of the VAE objective function, the ELBO. They find that β > 1 is required to improve the disentanglement of their latent representations.Interestingly, β > 1 corresponds analytically to −1 < m < 0 in Equation 9, since the overall normalization of the objective function does not impact the location of its extrema. That is, Equation 9 is equivalent to the β-VAE objective function with β = (1 + m) −1 .Since m in AutoGen represents the number of times a high-fidelity reconstruction is demanded (in addition to a single generation from the prior), β-VAE with β > 1 is analytically equivalent to demanding a negative number of high-fidelity reconstructions. As an analytic function of m, with larger m corresponding to higher-fidelity reconstructions, negative m would correspond to a deprecation of the reconstruction quality. This is indeed what the authors in BID4 find and discuss. They view β-VAE as a technique to trade off more disentangled representations at the cost of lower-fidelity reconstructions, in contrast to our view of AutoGen as a technique to trade off higher-fidelity reconstructions at the cost of slightly inferior generation from the prior.In connecting to β-VAE, we have considered AutoGen with m as a real number. Practically, m could take positive real values, and can be seen as a hyperparameter that requires taskspecific tuning. From our results, we expect m ≈ 1 to be a useful ballpark value, with smaller m improving generation from the prior, and larger m improving reconstruction fidelity. The advantage of tuning m as described is that it has a principled interpretation at integer values; namely that of demanding m exact reconstructions from the latent, as derived in Section 2.In this light, KL annealing amounts to starting with m = ∞ at the beginning, and smoothly reducing m down to 0 during training. Thus, it is equivalent to optimizing the AutoGen lower bound given in Equation 9 with varying m during training. However, AutoGen should never require KL annealing.Scaling of the ELBO is common in multimodal generation, where the reconstruction terms are typically of different orders of magnitude BID14 BID15 . AutoGen can be adapted to provide a bound on a meaningful objective function in multimodal generation with well-scaled terms, by requiring a larger number of reconstructions for one data modality than the other. Autogen thus has broader applications in generative modelling, which the authors leave to future work. In this paper, we introduced AutoGen: a novel modelling approach to improve the descriptiveness of latent variables in generative models, by combining the log likelihood of m high-fidelity reconstructions via a stochastic autoencoder, with the log likelihood of a VAE. This approach is theoretically principled in that it retains a bound on a meaningful objective, and computationally amounts to a simple factor of (1 + m) in front of the reconstruction term in the standard ELBO. We find that the most natural version of AutoGen (with m = 1) provides significantly better reconstructions than the VAE approach to language modelling, and only minimally deprecates generation from the prior.
This paper introduces a novel generative modelling framework that avoids latent-variable collapse and clarifies the use of certain ad-hoc factors in training Variational Autoencoders. The paper proposes to resolve the issue about a variational auto-encoder ignoring the latent variables. This paper proposes adding a stochastic autoencoder to the original VAE model to address the problem that the LSTM decoder of a language model might be too strong to ignore the latent variable's information. This paper presents AutoGen, which combines a generative variational autoencoder with a high-fidelity reconstruction model based on autoencoder to better utiliza latent representation