Stable and Causal Inference for Discriminative Self-supervised Deep Visual Representations
This work addresses instability issues in self-supervised learning for visual tasks, offering a more efficient inference-time approach that could benefit researchers and practitioners in computer vision.
The paper tackled the problem of instability in discriminative self-supervised visual representations, which can hinder downstream performance, by analyzing these methods from a causal perspective and proposing inference-time solutions that improve stability and efficiency.
In recent years, discriminative self-supervised methods have made significant strides in advancing various visual tasks. The central idea of learning a data encoder that is robust to data distortions/augmentations is straightforward yet highly effective. Although many studies have demonstrated the empirical success of various learning methods, the resulting learned representations can exhibit instability and hinder downstream performance. In this study, we analyze discriminative self-supervised methods from a causal perspective to explain these unstable behaviors and propose solutions to overcome them. Our approach draws inspiration from prior works that empirically demonstrate the ability of discriminative self-supervised methods to demix ground truth causal sources to some extent. Unlike previous work on causality-empowered representation learning, we do not apply our solutions during the training process but rather during the inference process to improve time efficiency. Through experiments on both controlled image datasets and realistic image datasets, we show that our proposed solutions, which involve tempering a linear transformation with controlled synthetic data, are effective in addressing these issues.