Multiplicative update rules for accelerating deep learning training and
  increasing robustness
Manos Kirtas Nikolaos Passalis Anastasios Tefas
Even nowadays, where Deep Learning (DL) has achieved state-of-the-art
performance in a wide range of research domains, accelerating training and
building robust DL models remains a challenging task. To this end, generations
of researchers have pursued to develop robust methods for training DL
architectures that can be less sensitive to weight distributions, model
architectures and loss landscapes. However, such methods are limited to
adaptive learning rate optimizers, initialization schemes, and clipping
gradients without investigating the fundamental rule of parameters update.
Although multiplicative updates have contributed significantly to the early
development of machine learning and hold strong theoretical claims, to best of
our knowledge, this is the first work that investigate them in context of DL
training acceleration and robustness. In this work, we propose an optimization
framework that fits to a wide range of optimization algorithms and enables one
to apply alternative update rules. To this end, we propose a novel
multiplicative update rule and we extend their capabilities by combining it
with a traditional additive update term, under a novel hybrid update method. We
claim that the proposed framework accelerates training, while leading to more
robust models in contrast to traditionally used additive update rule and we
experimentally demonstrate their effectiveness in a wide range of task and
optimization methods. Such tasks ranging from convex and non-convex
optimization to difficult image classification benchmarks applying a wide range
of traditionally used optimization methods and Deep Neural Network (DNN)
architectures.