Investigation into the Training Dynamics of Learned Optimizers
Jan Sobotka Petr Šimánek Daniel Vašata
Optimization is an integral part of modern deep learning. Recently, the
concept of learned optimizers has emerged as a way to accelerate this
optimization process by replacing traditional, hand-crafted algorithms with
meta-learned functions. Despite the initial promising results of these methods,
issues with stability and generalization still remain, limiting their practical
use. Moreover, their inner workings and behavior under different conditions are
not yet fully understood, making it difficult to come up with improvements. For
this reason, our work examines their optimization trajectories from the
perspective of network architecture symmetries and parameter update
distributions. Furthermore, by contrasting the learned optimizers with their
manually designed counterparts, we identify several key insights that
demonstrate how each approach can benefit from the strengths of the other.