Rethinking The Uniformity Metric in Self-Supervised Learning
Xianghong Fang Jian Li Qiang Sun Benyou Wang
Uniformity plays a crucial role in the assessment of learned representations,
contributing to a deeper comprehension of self-supervised learning. The seminal
work by \citet{Wang2020UnderstandingCR} introduced a uniformity metric that
quantitatively measures the collapse degree of learned representations.
Directly optimizing this metric together with alignment proves to be effective
in preventing constant collapse. However, we present both theoretical and
empirical evidence revealing that this metric lacks sensitivity to dimensional
collapse, highlighting its limitations. To address this limitation and design a
more effective uniformity metric, this paper identifies five fundamental
properties, some of which the existing uniformity metric fails to meet. We
subsequently introduce a novel uniformity metric that satisfies all of these
desiderata and exhibits sensitivity to dimensional collapse. When applied as an
auxiliary loss in various established self-supervised methods, our proposed
uniformity metric consistently enhances their performance in downstream
tasks.Our code was released at
https://github.com/sunset-clouds/WassersteinUniformityMetric.