Derivation of Coordinate Descent Algorithms from Optimal Control Theory
I. M. Ross
Recently, it was posited that disparate optimization algorithms may be
coalesced in terms of a central source emanating from optimal control theory.
Here we further this proposition by showing how coordinate descent algorithms
may be derived from this emerging new principle. In particular, we show that
basic coordinate descent algorithms can be derived using a maximum principle
and a collection of max functions as "control" Lyapunov functions. The
convergence of the resulting coordinate descent algorithms is thus connected to
the controlled dissipation of their corresponding Lyapunov functions. The
operational metric for the search vector in all cases is given by the Hessian
of the convex objective function.