Universal Majorization-Minimization Algorithms
Matthew Streeter
Majorization-minimization (MM) is a family of optimization methods that
iteratively reduce a loss by minimizing a locally-tight upper bound, called a
majorizer. Traditionally, majorizers were derived by hand, and MM was only
applicable to a small number of well-studied problems. We present optimizers
that instead derive majorizers automatically, using a recent generalization of
Taylor mode automatic differentiation. These universal MM optimizers can be
applied to arbitrary problems and converge from any starting point, with no
hyperparameter tuning.