Symmetry Leads to Structured Constraint of Learning
Liu Ziyin
Due to common architecture designs, symmetries exist extensively in
contemporary neural networks. In this work, we unveil the importance of the
loss function symmetries in affecting, if not deciding, the learning behavior
of machine learning models. We prove that every mirror symmetry of the loss
function leads to a structured constraint, which becomes a favored solution
when either the weight decay or gradient noise is large. As direct corollaries,
we show that rescaling symmetry leads to sparsity, rotation symmetry leads to
low rankness, and permutation symmetry leads to homogeneous ensembling. Then,
we show that the theoretical framework can explain the loss of plasticity and
various collapse phenomena in neural networks and suggest how symmetries can be
used to design algorithms to enforce hard constraints in a differentiable way.