TransFusion: Contrastive Learning with Transformers
Huanran Li Daniel Pimentel-Alarcón
This paper proposes a novel framework, TransFusion, designed to make the
process of contrastive learning more analytical and explainable. TransFusion
consists of attention blocks whose softmax being replaced by ReLU, and its
final block's weighted-sum operation is truncated to leave the adjacency matrix
as the output. The model is trained by minimizing the Jensen-Shannon Divergence
between its output and the target affinity matrix, which indicates whether each
pair of samples belongs to the same or different classes. The main contribution
of TransFusion lies in defining a theoretical limit for answering two
fundamental questions in the field: the maximum level of data augmentation and
the minimum batch size required for effective contrastive learning.
Furthermore, experimental results indicate that TransFusion successfully
extracts features that isolate clusters from complex real-world data, leading
to improved classification accuracy in downstream tasks.