Quantifying the Variability Collapse of Neural Networks
Jing Xu Haoxiong Liu
Recent studies empirically demonstrate the positive relationship between the
transferability of neural networks and the within-class variation of the last
layer features. The recently discovered Neural Collapse (NC) phenomenon
provides a new perspective of understanding such last layer geometry of neural
networks. In this paper, we propose a novel metric, named Variability Collapse
Index (VCI), to quantify the variability collapse phenomenon in the NC
paradigm. The VCI metric is well-motivated and intrinsically related to the
linear probing loss on the last layer features. Moreover, it enjoys desired
theoretical and empirical properties, including invariance under invertible
linear transformations and numerical stability, that distinguishes it from
previous metrics. Our experiments verify that VCI is indicative of the
variability collapse and the transferability of pretrained neural networks.