Leader-Follower Neural Networks with Local Error Signals Inspired by
  Complex Collectives
Chenzhong Yin Mingxi Cheng Xiongye Xiao Xinghe Chen Shahin Nazarian Andrei Irimia Paul Bogdan
The collective behavior of a network with heterogeneous, resource-limited
information processing units (e.g., group of fish, flock of birds, or network
of neurons) demonstrates high self-organization and complexity. These emergent
properties arise from simple interaction rules where certain individuals can
exhibit leadership-like behavior and influence the collective activity of the
group. Motivated by the intricacy of these collectives, we propose a neural
network (NN) architecture inspired by the rules observed in nature's collective
ensembles. This NN structure contains workers that encompass one or more
information processing units (e.g., neurons, filters, layers, or blocks of
layers). Workers are either leaders or followers, and we train a
leader-follower neural network (LFNN) by leveraging local error signals and
optionally incorporating backpropagation (BP) and global loss. We investigate
worker behavior and evaluate LFNNs through extensive experimentation. Our LFNNs
trained with local error signals achieve significantly lower error rates than
previous BP-free algorithms on MNIST and CIFAR-10 and even surpass BP-enabled
baselines. In the case of ImageNet, our LFNN-l demonstrates superior
scalability and outperforms previous BP-free algorithms by a significant
margin.