Constrained Mean Shift Using Distant Yet Related Neighbors for Representation Learning
This addresses representation learning for computer vision by incrementally enhancing clustering methods with constrained neighbor selection.
The paper tackles the problem of representation learning by generalizing mean-shift to use distant yet semantically related neighbors, improving performance in self-supervised and semi-supervised settings. It outperforms MSF in SSL and PAWS in semi-supervised learning with fewer resources.
We are interested in representation learning in self-supervised, supervised, and semi-supervised settings. Some recent self-supervised learning methods like mean-shift (MSF) cluster images by pulling the embedding of a query image to be closer to its nearest neighbors (NNs). Since most NNs are close to the query by design, the averaging may not affect the embedding of the query much. On the other hand, far away NNs may not be semantically related to the query. We generalize the mean-shift idea by constraining the search space of NNs using another source of knowledge so that NNs are far from the query while still being semantically related. We show that our method (1) outperforms MSF in SSL setting when the constraint utilizes a different augmentation of an image from the previous epoch, and (2) outperforms PAWS in semi-supervised setting with less training resources when the constraint ensures that the NNs have the same pseudo-label as the query.