BP(λ): Online Learning via Synthetic Gradients
Joseph Pemberton Rui Ponte Costa
Training recurrent neural networks typically relies on backpropagation
through time (BPTT). BPTT depends on forward and backward passes to be
completed, rendering the network locked to these computations before loss
gradients are available. Recently, Jaderberg et al. proposed synthetic
gradients to alleviate the need for full BPTT. In their implementation
synthetic gradients are learned through a mixture of backpropagated gradients
and bootstrapped synthetic gradients, analogous to the temporal difference (TD)
algorithm in Reinforcement Learning (RL). However, as in TD learning, heavy use
of bootstrapping can result in bias which leads to poor synthetic gradient
estimates. Inspired by the accumulate $\mathrm{TD}(\lambda)$ in RL, we propose
a fully online method for learning synthetic gradients which avoids the use of
BPTT altogether: accumulate $BP(\lambda)$. As in accumulate
$\mathrm{TD}(\lambda)$, we show analytically that accumulate
$\mathrm{BP}(\lambda)$ can control the level of bias by using a mixture of
temporal difference errors and recursively defined eligibility traces. We next
demonstrate empirically that our model outperforms the original implementation
for learning synthetic gradients in a variety of tasks, and is particularly
suited for capturing longer timescales. Finally, building on recent work we
reflect on accumulate $\mathrm{BP}(\lambda)$ as a principle for learning in
biological circuits. In summary, inspired by RL principles we introduce an
algorithm capable of bias-free online learning via synthetic gradients.