A Mean-Field Analysis of Neural Gradient Descent-Ascent: Applications to
  Functional Conditional Moment Equations
Yuchen Zhu Yufeng Zhang Zhaoran Wang Zhuoran Yang Xiaohong Chen
We study minimax optimization problems defined over infinite-dimensional
function classes. In particular, we restrict the functions to the class of
overparameterized two-layer neural networks and study (i) the convergence of
the gradient descent-ascent algorithm and (ii) the representation learning of
the neural network. As an initial step, we consider the minimax optimization
problem stemming from estimating a functional equation defined by conditional
expectations via adversarial estimation, where the objective function is
quadratic in the functional space. For this problem, we establish convergence
under the mean-field regime by considering the continuous-time and
infinite-width limit of the optimization dynamics. Under this regime, gradient
descent-ascent corresponds to a Wasserstein gradient flow over the space of
probability measures defined over the space of neural network parameters. We
prove that the Wasserstein gradient flow converges globally to a stationary
point of the minimax objective at a $\mathcal{O}(T^{-1} + \alpha^{-1} ) $
sublinear rate, and additionally finds the solution to the functional equation
when the regularizer of the minimax objective is strongly convex. Here $T$
denotes the time and $\alpha$ is a scaling parameter of the neural network. In
terms of representation learning, our results show that the feature
representation induced by the neural networks is allowed to deviate from the
initial one by the magnitude of $\mathcal{O}(\alpha^{-1})$, measured in terms
of the Wasserstein distance. Finally, we apply our general results to concrete
examples including policy evaluation, nonparametric instrumental variable
regression, and asset pricing.