An Ordering of Divergences for Variational Inference with Factorized
  Gaussian Approximations
Charles C. Margossian Loucas Pillaud-Vivien Lawrence K. Saul
Given an intractable distribution $p$, the problem of variational inference
(VI) is to compute the best approximation $q$ from some more tractable family
$\mathcal{Q}$. Most commonly the approximation is found by minimizing a
Kullback-Leibler (KL) divergence. However, there exist other valid choices of
divergences, and when $\mathcal{Q}$ does not contain~$p$, each divergence
champions a different solution. We analyze how the choice of divergence affects
the outcome of VI when a Gaussian with a dense covariance matrix is
approximated by a Gaussian with a diagonal covariance matrix. In this setting
we show that different divergences can be \textit{ordered} by the amount that
their variational approximations misestimate various measures of uncertainty,
such as the variance, precision, and entropy. We also derive an impossibility
theorem showing that no two of these measures can be simultaneously matched by
a factorized approximation; hence, the choice of divergence informs which
measure, if any, is correctly estimated. Our analysis covers the KL divergence,
the R\'enyi divergences, and a score-based divergence that compares $\nabla\log
p$ and $\nabla\log q$. We empirically evaluate whether these orderings hold
when VI is used to approximate non-Gaussian distributions.