On the Fisher-Rao Gradient of the Evidence Lower Bound
Nihat Ay Jesse van Oostrum
This article studies the Fisher-Rao gradient, also referred to as the natural
gradient, of the evidence lower bound, the ELBO, which plays a crucial role
within the theory of the Variational Autonecoder, the Helmholtz Machine and the
Free Energy Principle. The natural gradient of the ELBO is related to the
natural gradient of the Kullback-Leibler divergence from a target distribution,
the prime objective function of learning. Based on invariance properties of
gradients within information geometry, conditions on the underlying model are
provided that ensure the equivalence of minimising the prime objective function
and the maximisation of the ELBO.