Can Forward Gradient Match Backpropagation?
Louis Fournier Stéphane Rivaud Eugene Belilovsky Michael Eickenberg Edouard Oyallon
Forward Gradients - the idea of using directional derivatives in forward
differentiation mode - have recently been shown to be utilizable for neural
network training while avoiding problems generally associated with
backpropagation gradient computation, such as locking and memorization
requirements. The cost is the requirement to guess the step direction, which is
hard in high dimensions. While current solutions rely on weighted averages over
isotropic guess vector distributions, we propose to strongly bias our gradient
guesses in directions that are much more promising, such as feedback obtained
from small, local auxiliary networks. For a standard computer vision neural
network, we conduct a rigorous study systematically covering a variety of
combinations of gradient targets and gradient guesses, including those
previously presented in the literature. We find that using gradients obtained
from a local loss as a candidate direction drastically improves on random noise
in Forward Gradient methods.