AA-DLADMM: An Accelerated ADMM-based Framework for Training Deep Neural
  Networks
Zeinab Ebrahimi Gustavo Batista Mohammad Deghat
Stochastic gradient descent (SGD) and its many variants are the widespread
optimization algorithms for training deep neural networks. However, SGD suffers
from inevitable drawbacks, including vanishing gradients, lack of theoretical
guarantees, and substantial sensitivity to input. The Alternating Direction
Method of Multipliers (ADMM) has been proposed to address these shortcomings as
an effective alternative to the gradient-based methods. It has been
successfully employed for training deep neural networks. However, ADMM-based
optimizers have a slow convergence rate. This paper proposes an Anderson
Acceleration for Deep Learning ADMM (AA-DLADMM) algorithm to tackle this
drawback. The main intention of the AA-DLADMM algorithm is to employ Anderson
acceleration to ADMM by considering it as a fixed-point iteration and attaining
a nearly quadratic convergence rate. We verify the effectiveness and efficiency
of the proposed AA-DLADMM algorithm by conducting extensive experiments on four
benchmark datasets contrary to other state-of-the-art optimizers.