Feature Normalization Prevents Collapse of Non-contrastive Learning
  Dynamics
Han Bao
Contrastive learning is a self-supervised representation learning framework,
where two positive views generated through data augmentation are made similar
by an attraction force in a data representation space, while a repulsive force
makes them far from negative examples. Non-contrastive learning, represented by
BYOL and SimSiam, further gets rid of negative examples and improves
computational efficiency. While learned representations may collapse into a
single point due to the lack of the repulsive force at first sight, Tian et al.
(2021) revealed through the learning dynamics analysis that the representations
can avoid collapse if data augmentation is sufficiently stronger than
regularization. However, their analysis does not take into account
commonly-used feature normalization, a normalizer before measuring the
similarity of representations, and hence excessively strong regularization may
collapse the dynamics, which is an unnatural behavior under the presence of
feature normalization. Therefore, we extend the previous theory based on the L2
loss by considering the cosine loss, which involves feature normalization. We
show that the cosine loss induces sixth-order dynamics (while the L2 loss
induces a third-order one), in which a stable equilibrium dynamically emerges
even if there are only collapsed solutions with given initial parameters. Thus,
we offer a new understanding that feature normalization plays an important role
in robustly preventing the dynamics collapse.