Supervised Contrastive Representation Learning: Landscape Analysis with
  Unconstrained Features
Tina Behnia Christos Thrampoulidis
Recent findings reveal that over-parameterized deep neural networks, trained
beyond zero training-error, exhibit a distinctive structural pattern at the
final layer, termed as Neural-collapse (NC). These results indicate that the
final hidden-layer outputs in such networks display minimal within-class
variations over the training set. While existing research extensively
investigates this phenomenon under cross-entropy loss, there are fewer studies
focusing on its contrastive counterpart, supervised contrastive (SC) loss.
Through the lens of NC, this paper employs an analytical approach to study the
solutions derived from optimizing the SC loss. We adopt the unconstrained
features model (UFM) as a representative proxy for unveiling NC-related
phenomena in sufficiently over-parameterized deep networks. We show that,
despite the non-convexity of SC loss minimization, all local minima are global
minima. Furthermore, the minimizer is unique (up to a rotation). We prove our
results by formalizing a tight convex relaxation of the UFM. Finally, through
this convex formulation, we delve deeper into characterizing the properties of
global solutions under label-imbalanced training data.