Towards Demystifying the Generalization Behaviors When Neural Collapse
  Emerges
Peifeng Gao Qianqian Xu Yibo Yang Peisong Wen Huiyang Shao Zhiyong Yang Bernard Ghanem Qingming Huang
Neural Collapse (NC) is a well-known phenomenon of deep neural networks in
the terminal phase of training (TPT). It is characterized by the collapse of
features and classifier into a symmetrical structure, known as simplex
equiangular tight frame (ETF). While there have been extensive studies on
optimization characteristics showing the global optimality of neural collapse,
little research has been done on the generalization behaviors during the
occurrence of NC. Particularly, the important phenomenon of generalization
improvement during TPT has been remaining in an empirical observation and
lacking rigorous theoretical explanation. In this paper, we establish the
connection between the minimization of CE and a multi-class SVM during TPT, and
then derive a multi-class margin generalization bound, which provides a
theoretical explanation for why continuing training can still lead to accuracy
improvement on test set, even after the train accuracy has reached 100%.
Additionally, our further theoretical results indicate that different alignment
between labels and features in a simplex ETF can result in varying degrees of
generalization improvement, despite all models reaching NC and demonstrating
similar optimization performance on train set. We refer to this newly
discovered property as "non-conservative generalization". In experiments, we
also provide empirical observations to verify the indications suggested by our
theoretical results.