Understanding Predictive Coding as an Adaptive Trust-Region Method
Francesco Innocenti Ryan Singh Christopher L. Buckley
Predictive coding (PC) is a brain-inspired local learning algorithm that has
recently been suggested to provide advantages over backpropagation (BP) in
biologically relevant scenarios. While theoretical work has mainly focused on
showing how PC can approximate BP in various limits, the putative benefits of
"natural" PC are less understood. Here we develop a theory of PC as an adaptive
trust-region (TR) algorithm that uses second-order information. We show that
the learning dynamics of PC can be interpreted as interpolating between BP's
loss gradient direction and a TR direction found by the PC inference dynamics.
Our theory suggests that PC should escape saddle points faster than BP, a
prediction which we prove in a shallow linear model and support with
experiments on deeper networks. This work lays a foundation for understanding
PC in deep and wide networks.