EnSiam: Self-Supervised Learning With Ensemble Representations
Kyoungmin Han Minsik Lee
Recently, contrastive self-supervised learning, where the proximity of
representations is determined based on the identities of samples, has made
remarkable progress in unsupervised representation learning. SimSiam is a
well-known example in this area, known for its simplicity yet powerful
performance. However, it is known to be sensitive to changes in training
configurations, such as hyperparameters and augmentation settings, due to its
structural characteristics. To address this issue, we focus on the similarity
between contrastive learning and the teacher-student framework in knowledge
distillation. Inspired by the ensemble-based knowledge distillation approach,
the proposed method, EnSiam, aims to improve the contrastive learning procedure
using ensemble representations. This can provide stable pseudo labels,
providing better performance. Experiments demonstrate that EnSiam outperforms
previous state-of-the-art methods in most cases, including the experiments on
ImageNet, which shows that EnSiam is capable of learning high-quality
representations.