Adam-family Methods with Decoupled Weight Decay in Deep Learning
Kuangyu Ding Nachuan Xiao Kim-Chuan Toh
In this paper, we investigate the convergence properties of a wide class of
Adam-family methods for minimizing quadratically regularized nonsmooth
nonconvex optimization problems, especially in the context of training
nonsmooth neural networks with weight decay. Motivated by the AdamW method, we
propose a novel framework for Adam-family methods with decoupled weight decay.
Within our framework, the estimators for the first-order and second-order
moments of stochastic subgradients are updated independently of the weight
decay term. Under mild assumptions and with non-diminishing stepsizes for
updating the primary optimization variables, we establish the convergence
properties of our proposed framework. In addition, we show that our proposed
framework encompasses a wide variety of well-known Adam-family methods, hence
offering convergence guarantees for these methods in the training of nonsmooth
neural networks. More importantly, we show that our proposed framework
asymptotically approximates the SGD method, thereby providing an explanation
for the empirical observation that decoupled weight decay enhances
generalization performance for Adam-family methods. As a practical application
of our proposed framework, we propose a novel Adam-family method named Adam
with Decoupled Weight Decay (AdamD), and establish its convergence properties
under mild conditions. Numerical experiments demonstrate that AdamD outperforms
Adam and is comparable to AdamW, in the aspects of both generalization
performance and efficiency.