Layer Collaboration in the Forward-Forward Algorithm
Guy Lorberbom Itai Gat Yossi Adi Alex Schwing Tamir Hazan
Backpropagation, which uses the chain rule, is the de-facto standard
algorithm for optimizing neural networks nowadays. Recently, Hinton (2022)
proposed the forward-forward algorithm, a promising alternative that optimizes
neural nets layer-by-layer, without propagating gradients throughout the
network. Although such an approach has several advantages over back-propagation
and shows promising results, the fact that each layer is being trained
independently limits the optimization process. Specifically, it prevents the
network's layers from collaborating to learn complex and rich features. In this
work, we study layer collaboration in the forward-forward algorithm. We show
that the current version of the forward-forward algorithm is suboptimal when
considering information flow in the network, resulting in a lack of
collaboration between layers of the network. We propose an improved version
that supports layer collaboration to better utilize the network structure,
while not requiring any additional assumptions or computations. We empirically
demonstrate the efficacy of the proposed version when considering both
information flow and objective metrics. Additionally, we provide a theoretical
motivation for the proposed method, inspired by functional entropy theory.