More Synergy, Less Redundancy: Exploiting Joint Mutual Information for Self-Supervised Learning
This work addresses a fundamental problem in self-supervised learning for improving model performance without data annotation, though it appears incremental as it builds on existing redundancy reduction baselines.
The paper tackles the unclear role of mutual information in self-supervised learning by reformulating it using partial information decomposition to minimize redundancy and maximize synergy between augmentation views, resulting in a new training protocol that shows effectiveness across multiple datasets and downstream tasks.
Self-supervised learning (SSL) is now a serious competitor for supervised learning, even though it does not require data annotation. Several baselines have attempted to make SSL models exploit information about data distribution, and less dependent on the augmentation effect. However, there is no clear consensus on whether maximizing or minimizing the mutual information between representations of augmentation views practically contribute to improvement or degradation in performance of SSL models. This paper is a fundamental work where, we investigate role of mutual information in SSL, and reformulate the problem of SSL in the context of a new perspective on mutual information. To this end, we consider joint mutual information from the perspective of partial information decomposition (PID) as a key step in \textbf{reliable multivariate information measurement}. PID enables us to decompose joint mutual information into three important components, namely, unique information, redundant information and synergistic information. Our framework aims for minimizing the redundant information between views and the desired target representation while maximizing the synergistic information at the same time. Our experiments lead to a re-calibration of two redundancy reduction baselines, and a proposal for a new SSL training protocol. Extensive experimental results on multiple datasets and two downstream tasks show the effectiveness of this framework.