More Synergy, Less Redundancy: Exploiting Joint Mutual Information for
  Self-Supervised Learning
Salman Mohamadi Gianfranco Doretto Donald A. Adjeroh
Self-supervised learning (SSL) is now a serious competitor for supervised
learning, even though it does not require data annotation. Several baselines
have attempted to make SSL models exploit information about data distribution,
and less dependent on the augmentation effect. However, there is no clear
consensus on whether maximizing or minimizing the mutual information between
representations of augmentation views practically contribute to improvement or
degradation in performance of SSL models. This paper is a fundamental work
where, we investigate role of mutual information in SSL, and reformulate the
problem of SSL in the context of a new perspective on mutual information. To
this end, we consider joint mutual information from the perspective of partial
information decomposition (PID) as a key step in \textbf{reliable multivariate
information measurement}. PID enables us to decompose joint mutual information
into three important components, namely, unique information, redundant
information and synergistic information. Our framework aims for minimizing the
redundant information between views and the desired target representation while
maximizing the synergistic information at the same time. Our experiments lead
to a re-calibration of two redundancy reduction baselines, and a proposal for a
new SSL training protocol. Extensive experimental results on multiple datasets
and two downstream tasks show the effectiveness of this framework.