Implicit regularisation in stochastic gradient descent: from
  single-objective to two-player games
Mihaela Rosca Marc Peter Deisenroth
Recent years have seen many insights on deep learning optimisation being
brought forward by finding implicit regularisation effects of commonly used
gradient-based optimisers. Understanding implicit regularisation can not only
shed light on optimisation dynamics, but it can also be used to improve
performance and stability across problem domains, from supervised learning to
two-player games such as Generative Adversarial Networks. An avenue for finding
such implicit regularisation effects has been quantifying the discretisation
errors of discrete optimisers via continuous-time flows constructed by backward
error analysis (BEA). The current usage of BEA is not without limitations,
since not all the vector fields of continuous-time flows obtained using BEA can
be written as a gradient, hindering the construction of modified losses
revealing implicit regularisers. In this work, we provide a novel approach to
use BEA, and show how our approach can be used to construct continuous-time
flows with vector fields that can be written as gradients. We then use this to
find previously unknown implicit regularisation effects, such as those induced
by multiple stochastic gradient descent steps while accounting for the exact
data batches used in the updates, and in generally differentiable two-player
games.