In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One way to address this question is to find an upper bound on the probability (equivalently a lower bound on the negative log likelihood) that the model can assign to some data as one varies the prior and/or the likelihood function in a latent variable model. The core of our contribution is to formally show that the problem of optimizing priors in latent variable models is exactly an instance of the variational optimization problem that information theorists solve when computing rate-distortion functions, and then to use this to derive a lower bound on negative log likelihood. Moreover, we will show that if changing the prior can improve the log likelihood, then there is a way to change the likelihood function instead and attain the same log likelihood, and thus rate-distortion theory is of relevance to both optimizing priors as well as optimizing likelihood functions. We will experimentally argue for the usefulness of quantities derived from rate-distortion theory in latent variable modeling by applying them to a problem in image modeling. A statistician plans to use a latent variable model DISPLAYFORM0 where p(z) is known as the prior over the latent variables, and (x|z) is the likelihood of the data conditional on the latent variables; we will often use (p(z), (x|z)) as a shorthand for the model. Frequently, both the prior and the likelihood are parametrized and the statistician's job is to find reasonable parametric families for both -an optimization algorithm then chooses the parameter within those families. The task of designing these parametric families can sometimes be time consuming -this is one of the key modeling tasks when one adopts the representation learning viewpoint in machine learning.In this article we ask the question of how much p(z) can be improved if one fixes (x|z) and viceversa, with the goal of equipping the statistician with tools to make decisions on where to invest her time. One way to answer whether p(z) can be improved for fixed (x|z) is to drop the assumption that p(z) must belong to a particular family and ask how a model could improve in an unrestricted setting. Mathematically, given data {x 1 , · · · , x N } the first problem we study is the following optimization problem: for a fixed (x|z), DISPLAYFORM1 which as we will show, is also indirectly connected to the problem of determining if (x|z) can be improved for a given fixed p(z). The quantity being optimized in (2) is called the average negative log likelihood of the data, and is used whenever one assumes that the data {x 1 , · · · , x N } have been drawn statistically independently at random. Note that in this paper we are overloading the meaning of p(z): in (1) it refers to prior in the "current" latent variable model in the statistician's hands, and in (2) and other similar equations, it refers to a prior that can be optimized. We believe that the meaning will be clear depending on the context.Obviously, for any given (x|z), p(z), from the definition (1) we have the trivial upper bound DISPLAYFORM2 Can we give a good lower bound? A lower bound could tell us how far we can improve the model by changing the prior. The answer turns out to be in the affirmative. In an important paper, BID14 proved several facts about the problem of optimizing priors in latent variable models. In particular, he showed that DISPLAYFORM3 This result is very general -it holds for both discrete and continuous latent variable spaces, scalar or vector. It is also sharp -if you plug in the right prior, the upper and lower bounds match. It also has the advantage that the lower bound is written as a function of the trivial upper bound (3) -if someone proposes a latent variable model p(x) which uses a likelihood function (x|z), the optimal negative log likelihood value when we optimize the prior is thus known to be within a gap of DISPLAYFORM4 bits. The individual quantities under the sup c(z) DISPLAYFORM5 have a specific functional meaning: they can be regarded as multiplicative factors that tell you how to modify the prior to improve the log likelihood (see the Blahut-Arimoto algorithm BID3 ): DISPLAYFORM6 Lindsay (1983) derived his results with no apparent reference to earlier published work on ratedistortion theory, which is how information theorists study the problem of lossy compression BID19 . There is no reason for why this connection could be reasonably made, as it is not immediately obvious that the problems are connected; However, the quantity c(z) and the lower bound (4) in fact can be derived from ideas in Berger's book on rate-distortion theory BID2 ; in fact Lindsey's classical result that the optimal prior in a latent variable model has finite support with size equal to the size of the training data can be, drawing the appropriate connection, seen as a result of a similar result in rate-distortion theory also contained in BID2 .With time, the fundamental connection between the log likelihood optimization in latent variable modeling and the computation of rate-distortion functions became more clear. Although not explicitly mentioned in these terms, BID18 BID16 restate the optimal log likelihood as a problem of minimizing the variational free energy of a certain statistical system; this expression is identical to the one that is optimized in rate-distortion theory. The Information Bottleneck method of BID22 is a highly successful idea that exists in this boundary, having created a subfield of research that remains relevant nowadays BID23 ) BID20 . BID21 showed a fundamental connection between maximum likelihood and the information bottleneck method: "every fixed point of the IB-functional defines a fixed point of the (log) likelihood and vice versa". BID1 defined a rate-distortion problem where the output alphabet Z is finite and is jointly optimized with the test channel. By specializing to the case of where the distortion measure is a Bregman divergence, they showed the mathematical equivalence between this problem and that of maximum likelihood estimation where the likelihood function is an exponential family distribution derived from the Bregman divergence. BID27 study a variation of the maximum likelihood estimation for latent variable models where the maximum likelihood criterion is instead replaced with the entropic risk measure. The autoencoder concept extensively used in the neural network community is arguably directly motivated by the encoder/decoder concepts in lossy/lossless compression. BID7 proposed using a matrix based expression motivated by rate-distortion ideas in order to train autoencoders while avoiding estimating mutual information directly. Recently, BID0 exploited the β-VAE loss function of BID8 to explicitly introduce a trade-off between rate and distortion in the latent variable modeling problem, where the notions of rate and distortion have similarities to those used in our article.Latent Variable Modeling is undergoing an exciting moment as it is a form of representation learning, the latter having shown to be an important tool in reaching remarkable performance in difficult machine learning problems while simultaneously avoiding feature engineering. This prompted us to look deeper into rate-distortion theory as a tool for developing a theory of representation learning. What excites us is the possibility of using a large repertoire of tools, algorithms and theoretical results in lossy compression in meaningful ways in latent variable modeling; notably, we believe that beyond what we will state in this article, Shannon's famous lossy source coding theorem, the information theory of multiple senders and receivers, and the rich Shannon-style equalities and inequalities involving entropy and mutual information are all of fundamental relevance to the classical problem (1) and more complex variations of it.The goal of this article is laying down a firm foundation that we can use to build towards the program above. We start by proving the fundamental equivalence between these two fields by using simple convexity arguments, avoiding the variational calculus arguments that had been used before. We then take an alternate path to proving (2) also involving simple convexity arguments inspired by earlier results in rate-distortion theory.We then focus on what is a common problem -instead of improving a prior for a fixed likelihood function, improve the likelihood function for a fixed prior but for a fixed prior. Interestingly, ratedistortion theory still is relevant to this problem, although the question that we are able to answer with it is smaller in scope. Through a simple change of variable argument , we will argue that if the negative log likelihood can be improved by modifying the prior, exactly the same negative log likelihood can be attained by modifying the likelihood function instead. Thus if rate-distortion theory predicts that there is scope for improvement for a prior, the same holds for the likelihood function but conversely, while rate-distortion theory can precisely determine when it is that a prior can no longer be improved, the same cannot be said for the likelihood function.Finally, we test whether the lower bound derived and the corresponding fundamental quantity c(z) are useful in practice when making modeling decisions by applying these ideas to a problem in image modeling for which there have been several recent results involving Variational Autoencoders. The main goal for this article was to argue strongly for the inclusion of rate-distortion theory as key for developing a theory of representation learning. In the article we showed how some classical results in latent variable modeling can be seen as relatively simple consequences of results in ratedistortion function computation, and further argued that these results help in understanding whether prior or likelihood functions can be improved further (the latter with some limitations), demonstrating this with some experimental results in an image modeling problem. There is a large repertoire of tools, algorithms and theoretical results in lossy compression that we believe can be applied in meaningful ways to latent variable modeling. For example, while rate-distortion function computation is an important subject in information theory, the true crown jewel is Shannon's famous source coding theorem; to-date we are not aware of this important result being connected directly to the problem of latent variable modeling. Similarly, rate-distortion theory has evolved since Shannon's original publication to treat multiple sources and sinks; we believe that these are of relevance in more complex modeling tasks. This research will be the subject of future work.
Use rate-distortion theory to bound how much a latent variable model can be improved Addresses problems of optimization of the prior in the latent variable model and the selection of the likelihood function by proposing criteria based on a lower-bound on the negative log-likelihood. Presents a theorem which gives a lower bound on negative log likelihood of rate-distortion for latent-variable modeling The authors argue that the rate-distortion theory for lossy compression provides a natural toolkit for studying latent variable models proposes a lower bound.