Contrastive Learning with Orthonormal Anchors (CLOA)
Huanran Li Daniel Pimentel-Alarcón
This study focuses on addressing the instability issues prevalent in
contrastive learning, specifically examining the InfoNCE loss function and its
derivatives. We reveal a critical observation that these loss functions exhibit
a restrictive behavior, leading to a convergence phenomenon where embeddings
tend to merge into a singular point. This "over-fusion" effect detrimentally
affects classification accuracy in subsequent supervised-learning tasks.
Through theoretical analysis, we demonstrate that embeddings, when equalized or
confined to a rank-1 linear subspace, represent a local minimum for InfoNCE. In
response to this challenge, our research introduces an innovative strategy that
leverages the same or fewer labeled data than typically used in the fine-tuning
phase. The loss we proposed, Orthonormal Anchor Regression Loss, is designed to
disentangle embedding clusters, significantly enhancing the distinctiveness of
each embedding while simultaneously ensuring their aggregation into dense,
well-defined clusters. Our method demonstrates remarkable improvements with
just a fraction of the conventional label requirements, as evidenced by our
results on CIFAR10 and CIFAR100 datasets.