AdamL: A fast adaptive gradient method incorporating loss function
Lu Xia Stefano Massei
Adaptive first-order optimizers are fundamental tools in deep learning,
although they may suffer from poor generalization due to the nonuniform
gradient scaling. In this work, we propose AdamL, a novel variant of the Adam
optimizer, that takes into account the loss function information to attain
better generalization results. We provide sufficient conditions that together
with the Polyak-Lojasiewicz inequality, ensure the linear convergence of AdamL.
As a byproduct of our analysis, we prove similar convergence properties for the
EAdam, and AdaBelief optimizers. Experimental results on benchmark functions
show that AdamL typically achieves either the fastest convergence or the lowest
objective function values when compared to Adam, EAdam, and AdaBelief. These
superior performances are confirmed when considering deep learning tasks such
as training convolutional neural networks, training generative adversarial
networks using vanilla convolutional neural networks, and long short-term
memory networks. Finally, in the case of vanilla convolutional neural networks,
AdamL stands out from the other Adam's variants and does not require the manual
adjustment of the learning rate during the later stage of the training.