Intelligent gradient amplification for deep neural networks
Sunitha Basodi Krishna Pusuluri Xueli Xiao Yi Pan
Deep learning models offer superior performance compared to other machine
learning techniques for a variety of tasks and domains, but pose their own
challenges. In particular, deep learning models require larger training times
as the depth of a model increases, and suffer from vanishing gradients. Several
solutions address these problems independently, but there have been minimal
efforts to identify an integrated solution that improves the performance of a
model by addressing vanishing gradients, as well as accelerates the training
process to achieve higher performance at larger learning rates. In this work,
we intelligently determine which layers of a deep learning model to apply
gradient amplification to, using a formulated approach that analyzes gradient
fluctuations of layers during training. Detailed experiments are performed for
simpler and deeper neural networks using two different intelligent measures and
two different thresholds that determine the amplification layers, and a
training strategy where gradients are amplified only during certain epochs.
Results show that our amplification offers better performance compared to the
original models, and achieves accuracy improvement of around 2.5% on CIFAR- 10
and around 4.5% on CIFAR-100 datasets, even when the models are trained with
higher learning rates.