Adaptive Similarity Bootstrapping for Self-Distillation based Representation Learning
This addresses a specific problem in self-supervised learning for researchers, offering an incremental improvement to existing self-distillation methods.
The paper tackles the performance drop or collapse when using nearest neighbor bootstrapping in self-distillation for representation learning, proposing an adaptive method based on latent space quality that leads to consistent improvements over naive bootstrapping and original baselines.
Most self-supervised methods for representation learning leverage a cross-view consistency objective i.e., they maximize the representation similarity of a given image's augmented views. Recent work NNCLR goes beyond the cross-view paradigm and uses positive pairs from different images obtained via nearest neighbor bootstrapping in a contrastive setting. We empirically show that as opposed to the contrastive learning setting which relies on negative samples, incorporating nearest neighbor bootstrapping in a self-distillation scheme can lead to a performance drop or even collapse. We scrutinize the reason for this unexpected behavior and provide a solution. We propose to adaptively bootstrap neighbors based on the estimated quality of the latent space. We report consistent improvements compared to the naive bootstrapping approach and the original baselines. Our approach leads to performance improvements for various self-distillation method/backbone combinations and standard downstream tasks. Our code is publicly available at https://github.com/tileb1/AdaSim.