Structural Credit Assignment with Coordinated Exploration
Stephen Chung
A biologically plausible method for training an Artificial Neural Network
(ANN) involves treating each unit as a stochastic Reinforcement Learning (RL)
agent, thereby considering the network as a team of agents. Consequently, all
units can learn via REINFORCE, a local learning rule modulated by a global
reward signal, which aligns more closely with biologically observed forms of
synaptic plasticity. However, this learning method tends to be slow and does
not scale well with the size of the network. This inefficiency arises from two
factors impeding effective structural credit assignment: (i) all units
independently explore the network, and (ii) a single reward is used to evaluate
the actions of all units. Accordingly, methods aimed at improving structural
credit assignment can generally be classified into two categories. The first
category includes algorithms that enable coordinated exploration among units,
such as MAP propagation. The second category encompasses algorithms that
compute a more specific reward signal for each unit within the network, like
Weight Maximization and its variants. In this research report, our focus is on
the first category. We propose the use of Boltzmann machines or a recurrent
network for coordinated exploration. We show that the negative phase, which is
typically necessary to train Boltzmann machines, can be removed. The resulting
learning rules are similar to the reward-modulated Hebbian learning rule.
Experimental results demonstrate that coordinated exploration significantly
exceeds independent exploration in training speed for multiple stochastic and
discrete units based on REINFORCE, even surpassing straight-through estimator
(STE) backpropagation.