CLCE: An Approach to Refining Cross-Entropy and Contrastive Learning for
  Optimized Learning Fusion
Zijun Long George Killick Lipeng Zhuang Gerardo Aragon-Camarasa Zaiqiao Meng Richard Mccreadie
State-of-the-art pre-trained image models predominantly adopt a two-stage
approach: initial unsupervised pre-training on large-scale datasets followed by
task-specific fine-tuning using Cross-Entropy loss~(CE). However, it has been
demonstrated that CE can compromise model generalization and stability. While
recent works employing contrastive learning address some of these limitations
by enhancing the quality of embeddings and producing better decision
boundaries, they often overlook the importance of hard negative mining and rely
on resource intensive and slow training using large sample batches. To counter
these issues, we introduce a novel approach named CLCE, which integrates
Label-Aware Contrastive Learning with CE. Our approach not only maintains the
strengths of both loss functions but also leverages hard negative mining in a
synergistic way to enhance performance. Experimental results demonstrate that
CLCE significantly outperforms CE in Top-1 accuracy across twelve benchmarks,
achieving gains of up to 3.52% in few-shot learning scenarios and 3.41% in
transfer learning settings with the BEiT-3 model. Importantly, our proposed
CLCE approach effectively mitigates the dependency of contrastive learning on
large batch sizes such as 4096 samples per batch, a limitation that has
previously constrained the application of contrastive learning in
budget-limited hardware environments.