Hallmarks of Optimization Trajectories in Neural Networks and LLMs: The
  Lengths, Bends, and Dead Ends
Sidak Pal Singh Bobby He Thomas Hofmann Bernhard Schölkopf
We propose a fresh take on understanding the mechanisms of neural networks by
analyzing the rich structure of parameters contained within their optimization
trajectories. Towards this end, we introduce some natural notions of the
complexity of optimization trajectories, both qualitative and quantitative,
which reveal the inherent nuance and interplay involved between various
optimization choices, such as momentum, weight decay, and batch size. We use
them to provide key hallmarks about the nature of optimization in deep neural
networks: when it goes right, and when it finds itself in a dead end. Further,
thanks to our trajectory perspective, we uncover an intertwined behaviour of
momentum and weight decay that promotes directional exploration, as well as a
directional regularization behaviour of some others. We perform experiments
over large-scale vision and language settings, including large language models
(LLMs) with up to 12 billion parameters, to demonstrate the value of our
approach.