Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training
Xi Chen Chang Gao Zuowen Wang Longbiao Cheng Sheng Zhou Shih-Chii Liu Tobi Delbruck
Recurrent Neural Networks (RNNs) are useful in temporal sequence tasks.
However, training RNNs involves dense matrix multiplications which require
hardware that can support a large number of arithmetic operations and memory
accesses. Implementing online training of RNNs on the edge calls for optimized
algorithms for an efficient deployment on hardware. Inspired by the spiking
neuron model, the Delta RNN exploits temporal sparsity during inference by
skipping over the update of hidden states from those inactivated neurons whose
change of activation across two timesteps is below a defined threshold. This
work describes a training algorithm for Delta RNNs that exploits temporal
sparsity in the backward propagation phase to reduce computational requirements
for training on the edge. Due to the symmetric computation graphs of forward
and backward propagation during training, the gradient computation of
inactivated neurons can be skipped. Results show a reduction of $\sim$80% in
matrix operations for training a 56k parameter Delta LSTM on the Fluent Speech
Commands dataset with negligible accuracy loss. Logic simulations of a hardware
accelerator designed for the training algorithm show 2-10X speedup in matrix
computations for an activation sparsity range of 50%-90%. Additionally, we show
that the proposed Delta RNN training will be useful for online incremental
learning on edge devices with limited computing resources.