SDEs for Minimax Optimization
Enea Monzio Compagnoni Antonio Orvieto Hans Kersting Frank Norbert Proske Aurelien Lucchi
Minimax optimization problems have attracted a lot of attention over the past
few years, with applications ranging from economics to machine learning. While
advanced optimization methods exist for such problems, characterizing their
dynamics in stochastic scenarios remains notably challenging. In this paper, we
pioneer the use of stochastic differential equations (SDEs) to analyze and
compare Minimax optimizers. Our SDE models for Stochastic Gradient
Descent-Ascent, Stochastic Extragradient, and Stochastic Hamiltonian Gradient
Descent are provable approximations of their algorithmic counterparts, clearly
showcasing the interplay between hyperparameters, implicit regularization, and
implicit curvature-induced noise. This perspective also allows for a unified
and simplified analysis strategy based on the principles of It\^o calculus.
Finally, our approach facilitates the derivation of convergence conditions and
closed-form solutions for the dynamics in simplified settings, unveiling
further insights into the behavior of different optimizers.