Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent
Frederik Köhne Leonie Kreis Anton Schiela Roland Herzog
This paper proposes a novel approach to adaptive step sizes in stochastic
gradient descent (SGD) by utilizing quantities that we have identified as
numerically traceable -- the Lipschitz constant for gradients and a concept of
the local variance in search directions. Our findings yield a nearly
hyperparameter-free algorithm for stochastic optimization, which has provable
convergence properties when applied to quadratic problems and exhibits truly
problem adaptive behavior on classical image classification tasks. Our
framework enables the potential inclusion of a preconditioner, thereby enabling
the implementation of adaptive step sizes for stochastic second-order
optimization methods.