On the Convergence of Coordinate Ascent Variational Inference
Anirban Bhattacharya Debdeep Pati Yun Yang
As a computational alternative to Markov chain Monte Carlo approaches,
variational inference (VI) is becoming more and more popular for approximating
intractable posterior distributions in large-scale Bayesian models due to its
comparable efficacy and superior efficiency. Several recent works provide
theoretical justifications of VI by proving its statistical optimality for
parameter estimation under various settings; meanwhile, formal analysis on the
algorithmic convergence aspects of VI is still largely lacking. In this paper,
we consider the common coordinate ascent variational inference (CAVI) algorithm
for implementing the mean-field (MF) VI towards optimizing a Kullback--Leibler
divergence objective functional over the space of all factorized distributions.
Focusing on the two-block case, we analyze the convergence of CAVI by
leveraging the extensive toolbox from functional analysis and optimization. We
provide general conditions for certifying global or local exponential
convergence of CAVI. Specifically, a new notion of generalized correlation for
characterizing the interaction between the constituting blocks in influencing
the VI objective functional is introduced, which according to the theory,
quantifies the algorithmic contraction rate of two-block CAVI. As
illustrations, we apply the developed theory to a number of examples, and
derive explicit problem-dependent upper bounds on the algorithmic contraction
rate.