Modified Step Size for Enhanced Stochastic Gradient Descent: Convergence
  and Experiments
M. Soheil Shamaee S. Fathi Hafshejani
This paper introduces a novel approach to enhance the performance of the
stochastic gradient descent (SGD) algorithm by incorporating a modified decay
step size based on $\frac{1}{\sqrt{t}}$. The proposed step size integrates a
logarithmic term, leading to the selection of smaller values in the final
iterations. Our analysis establishes a convergence rate of $O(\frac{\ln
T}{\sqrt{T}})$ for smooth non-convex functions without the
Polyak-{\L}ojasiewicz condition. To evaluate the effectiveness of our approach,
we conducted numerical experiments on image classification tasks using the
FashionMNIST, and CIFAR10 datasets, and the results demonstrate significant
improvements in accuracy, with enhancements of $0.5\%$ and $1.4\%$ observed,
respectively, compared to the traditional $\frac{1}{\sqrt{t}}$ step size. The
source code can be found at \\\url{https://github.com/Shamaeem/LNSQRTStepSize}.