Energy-based learning algorithms for analog computing: a comparative
  study
Benjamin Scellier Maxence Ernoult Jack Kendall Suhas Kumar
Energy-based learning algorithms have recently gained a surge of interest due
to their compatibility with analog (post-digital) hardware. Existing algorithms
include contrastive learning (CL), equilibrium propagation (EP) and coupled
learning (CpL), all consisting in contrasting two states, and differing in the
type of perturbation used to obtain the second state from the first one.
However, these algorithms have never been explicitly compared on equal footing
with same models and datasets, making it difficult to assess their scalability
and decide which one to select in practice. In this work, we carry out a
comparison of seven learning algorithms, namely CL and different variants of EP
and CpL depending on the signs of the perturbations. Specifically, using these
learning algorithms, we train deep convolutional Hopfield networks (DCHNs) on
five vision tasks (MNIST, F-MNIST, SVHN, CIFAR-10 and CIFAR-100). We find that,
while all algorithms yield comparable performance on MNIST, important
differences in performance arise as the difficulty of the task increases. Our
key findings reveal that negative perturbations are better than positive ones,
and highlight the centered variant of EP (which uses two perturbations of
opposite sign) as the best-performing algorithm. We also endorse these findings
with theoretical arguments. Additionally, we establish new SOTA results with
DCHNs on all five datasets, both in performance and speed. In particular, our
DCHN simulations are 13.5 times faster with respect to Laborieux et al. (2021),
which we achieve thanks to the use of a novel energy minimisation algorithm
based on asynchronous updates, combined with reduced precision (16 bits).