Layer-wise Feedback Propagation
Leander Weber Jim Berend Alexander Binder Thomas Wiegand Wojciech Samek Sebastian Lapuschkin
In this paper, we present Layer-wise Feedback Propagation (LFP), a novel
training approach for neural-network-like predictors that utilizes
explainability, specifically Layer-wise Relevance Propagation(LRP), to assign
rewards to individual connections based on their respective contributions to
solving a given task. This differs from traditional gradient descent, which
updates parameters towards anestimated loss minimum. LFP distributes a reward
signal throughout the model without the need for gradient computations. It then
strengthens structures that receive positive feedback while reducingthe
influence of structures that receive negative feedback. We establish the
convergence of LFP theoretically and empirically, and demonstrate its
effectiveness in achieving comparable performance to gradient descent on
various models and datasets. Notably, LFP overcomes certain limitations
associated with gradient-based methods, such as reliance on meaningful
derivatives. We further investigate how the different LRP-rules can be extended
to LFP, what their effects are on training, as well as potential applications,
such as training models with no meaningful derivatives, e.g., step-function
activated Spiking Neural Networks (SNNs), or for transfer learning, to
efficiently utilize existing knowledge.