Probabilistic Self-supervised Learning via Scoring Rules Minimization
This work addresses representation learning challenges in self-supervised learning for computer vision applications, offering a novel probabilistic approach with theoretical backing.
The paper tackles the problem of improving representation quality and preventing collapse in self-supervised learning by proposing ProSMIN, a probabilistic method that uses scoring rule minimization and knowledge distillation between two networks, achieving superior accuracy and calibration on tasks like out-of-distribution detection and transfer learning on datasets such as ImageNet-O and ImageNet-C.
In this paper, we propose a novel probabilistic self-supervised learning via Scoring Rule Minimization (ProSMIN), which leverages the power of probabilistic models to enhance representation quality and mitigate collapsing representations. Our proposed approach involves two neural networks; the online network and the target network, which collaborate and learn the diverse distribution of representations from each other through knowledge distillation. By presenting the input samples in two augmented formats, the online network is trained to predict the target network representation of the same sample under a different augmented view. The two networks are trained via our new loss function based on proper scoring rules. We provide a theoretical justification for ProSMIN's convergence, demonstrating the strict propriety of its modified scoring rule. This insight validates the method's optimization process and contributes to its robustness and effectiveness in improving representation quality. We evaluate our probabilistic model on various downstream tasks, such as in-distribution generalization, out-of-distribution detection, dataset corruption, low-shot learning, and transfer learning. Our method achieves superior accuracy and calibration, surpassing the self-supervised baseline in a wide range of experiments on large-scale datasets like ImageNet-O and ImageNet-C, ProSMIN demonstrates its scalability and real-world applicability.