Dealing with unbounded gradients in stochastic saddle-point optimization
Gergely Neu Nneka Okolo
We study the performance of stochastic first-order methods for finding saddle
points of convex-concave functions. A notorious challenge faced by such methods
is that the gradients can grow arbitrarily large during optimization, which may
result in instability and divergence. In this paper, we propose a simple and
effective regularization technique that stabilizes the iterates and yields
meaningful performance guarantees even if the domain and the gradient noise
scales linearly with the size of the iterates (and is thus potentially
unbounded). Besides providing a set of general results, we also apply our
algorithm to a specific problem in reinforcement learning, where it leads to
performance guarantees for finding near-optimal policies in an average-reward
MDP without prior knowledge of the bias span.