Pushing Boundaries: Mixup's Influence on Neural Collapse
Quinn Fisher Haoming Meng Vardan Papyan
Mixup is a data augmentation strategy that employs convex combinations of
training instances and their respective labels to augment the robustness and
calibration of deep neural networks. Despite its widespread adoption, the
nuanced mechanisms that underpin its success are not entirely understood. The
observed phenomenon of Neural Collapse, where the last-layer activations and
classifier of deep networks converge to a simplex equiangular tight frame
(ETF), provides a compelling motivation to explore whether mixup induces
alternative geometric configurations and whether those could explain its
success. In this study, we delve into the last-layer activations of training
data for deep networks subjected to mixup, aiming to uncover insights into its
operational efficacy. Our investigation, spanning various architectures and
dataset pairs, reveals that mixup's last-layer activations predominantly
converge to a distinctive configuration different than one might expect. In
this configuration, activations from mixed-up examples of identical classes
align with the classifier, while those from different classes delineate
channels along the decision boundary. Moreover, activations in earlier layers
exhibit patterns, as if trained with manifold mixup. These findings are
unexpected, as mixed-up features are not simple convex combinations of feature
class means (as one might get, for example, by training mixup with the mean
squared error loss). By analyzing this distinctive geometric configuration, we
elucidate the mechanisms by which mixup enhances model calibration. To further
validate our empirical observations, we conduct a theoretical analysis under
the assumption of an unconstrained features model, utilizing the mixup loss.
Through this, we characterize and derive the optimal last-layer features under
the assumption that the classifier forms a simplex ETF.