Transferability of Representations Learned using Supervised Contrastive
  Learning Trained on a Multi-Domain Dataset
Alvin De Jun Tan Clement Tan Chai Kiat Yeo
Contrastive learning has shown to learn better quality representations than
models trained using cross-entropy loss. They also transfer better to
downstream datasets from different domains. However, little work has been done
to explore the transferability of representations learned using contrastive
learning when trained on a multi-domain dataset. In this paper, a study has
been conducted using the Supervised Contrastive Learning framework to learn
representations from the multi-domain DomainNet dataset and then evaluate the
transferability of the representations learned on other downstream datasets.
The fixed feature linear evaluation protocol will be used to evaluate the
transferability on 7 downstream datasets that were chosen across different
domains. The results obtained are compared to a baseline model that was trained
using the widely used cross-entropy loss. Empirical results from the
experiments showed that on average, the Supervised Contrastive Learning model
performed 6.05% better than the baseline model on the 7 downstream datasets.
The findings suggest that Supervised Contrastive Learning models can
potentially learn more robust representations that transfer better across
domains than cross-entropy models when trained on a multi-domain dataset.