Abide by the Law and Follow the Flow: Conservation Laws for Gradient
  Flows
Sibylle Marcotte Rémi Gribonval Gabriel Peyré
Understanding the geometric properties of gradient descent dynamics is a key
ingredient in deciphering the recent success of very large machine learning
models. A striking observation is that trained over-parameterized models retain
some properties of the optimization initialization. This "implicit bias" is
believed to be responsible for some favorable properties of the trained models
and could explain their good generalization properties. The purpose of this
article is threefold. First, we rigorously expose the definition and basic
properties of "conservation laws", which are maximal sets of independent
quantities conserved during gradient flows of a given model (e.g. of a ReLU
network with a given architecture) with any training data and any loss. Then we
explain how to find the exact number of these quantities by performing
finite-dimensional algebraic manipulations on the Lie algebra generated by the
Jacobian of the model. Finally, we provide algorithms (implemented in SageMath)
to: a) compute a family of polynomial laws; b) compute the number of (not
necessarily polynomial) conservation laws. We provide showcase examples that we
fully work out theoretically. Besides, applying the two algorithms confirms for
a number of ReLU network architectures that all known laws are recovered by the
algorithm, and that there are no other laws. Such computational tools pave the
way to understanding desirable properties of optimization initialization in
large machine learning models.