Neural Collapse for Cross-entropy Class-Imbalanced Learning with
  Unconstrained ReLU Feature Model
Hien Dang Tho Tran Tan Nguyen Nhat Ho
The current paradigm of training deep neural networks for classification
tasks includes minimizing the empirical risk that pushes the training loss
value towards zero, even after the training error has been vanished. In this
terminal phase of training, it has been observed that the last-layer features
collapse to their class-means and these class-means converge to the vertices of
a simplex Equiangular Tight Frame (ETF). This phenomenon is termed as Neural
Collapse (NC). To theoretically understand this phenomenon, recent works employ
a simplified unconstrained feature model to prove that NC emerges at the global
solutions of the training problem. However, when the training dataset is
class-imbalanced, some NC properties will no longer be true. For example, the
class-means geometry will skew away from the simplex ETF when the loss
converges. In this paper, we generalize NC to imbalanced regime for
cross-entropy loss under the unconstrained ReLU feature model. We prove that,
while the within-class features collapse property still holds in this setting,
the class-means will converge to a structure consisting of orthogonal vectors
with different lengths. Furthermore, we find that the classifier weights are
aligned to the scaled and centered class-means with scaling factors depend on
the number of training samples of each class, which generalizes NC in the
class-balanced setting. We empirically prove our results through experiments on
practical architectures and dataset.