An Automatic Learning Rate Schedule Algorithm for Achieving Faster
  Convergence and Steeper Descent
Zhao Song Chiwun Yang
The delta-bar-delta algorithm is recognized as a learning rate adaptation
technique that enhances the convergence speed of the training process in
optimization by dynamically scheduling the learning rate based on the
difference between the current and previous weight updates. While this
algorithm has demonstrated strong competitiveness in full data optimization
when compared to other state-of-the-art algorithms like Adam and SGD, it may
encounter convergence issues in mini-batch optimization scenarios due to the
presence of noisy gradients.
  In this study, we thoroughly investigate the convergence behavior of the
delta-bar-delta algorithm in real-world neural network optimization. To address
any potential convergence challenges, we propose a novel approach called RDBD
(Regrettable Delta-Bar-Delta). Our approach allows for prompt correction of
biased learning rate adjustments and ensures the convergence of the
optimization process. Furthermore, we demonstrate that RDBD can be seamlessly
integrated with any optimization algorithm and significantly improve the
convergence speed.
  By conducting extensive experiments and evaluations, we validate the
effectiveness and efficiency of our proposed RDBD approach. The results
showcase its capability to overcome convergence issues in mini-batch
optimization and its potential to enhance the convergence speed of various
optimization algorithms. This research contributes to the advancement of
optimization techniques in neural network training, providing practitioners
with a reliable automatic learning rate scheduler for achieving faster
convergence and improved optimization outcomes.