Stochastic Gradient Descent with Preconditioned Polyak Step-size
Farshed Abdukhakimov Chulu Xiang Dmitry Kamzolov Martin Takáč
Stochastic Gradient Descent (SGD) is one of the many iterative optimization
methods that are widely used in solving machine learning problems. These
methods display valuable properties and attract researchers and industrial
machine learning engineers with their simplicity. However, one of the
weaknesses of this type of methods is the necessity to tune learning rate
(step-size) for every loss function and dataset combination to solve an
optimization problem and get an efficient performance in a given time budget.
Stochastic Gradient Descent with Polyak Step-size (SPS) is a method that offers
an update rule that alleviates the need of fine-tuning the learning rate of an
optimizer. In this paper, we propose an extension of SPS that employs
preconditioning techniques, such as Hutchinson's method, Adam, and AdaGrad, to
improve its performance on badly scaled and/or ill-conditioned datasets.