Learning to optimize with convergence guarantees using nonlinear system
  theory
Andrea Martin Luca Furieri
The increasing reliance on numerical methods for controlling dynamical
systems and training machine learning models underscores the need to devise
algorithms that dependably and efficiently navigate complex optimization
landscapes. Classical gradient descent methods offer strong theoretical
guarantees for convex problems; however, they demand meticulous hyperparameter
tuning for non-convex ones. The emerging paradigm of learning to optimize (L2O)
automates the discovery of algorithms with optimized performance leveraging
learning models and data - yet, it lacks a theoretical framework to analyze
convergence and robustness of the learned algorithms. In this paper, we fill
this gap by harnessing nonlinear system theory. Specifically, we propose an
unconstrained parametrization of all convergent algorithms for smooth
non-convex objective functions. Notably, our framework is directly compatible
with automatic differentiation tools, ensuring convergence by design while
learning to optimize.