Exploring Set Similarity for Dense Self-supervised Representation Learning
This addresses noisy backgrounds in dense self-supervised learning for computer vision tasks, representing an incremental improvement over existing methods.
The paper tackles the problem of noisy pixel-level correspondences in dense self-supervised representation learning by proposing SetSim, which generalizes similarity learning from pixel-wise to set-wise to improve robustness. The method achieves superior performance to state-of-the-art methods on object detection, keypoint detection, instance segmentation, and semantic segmentation.
By considering the spatial correspondence, dense self-supervised representation learning has achieved superior performance on various dense prediction tasks. However, the pixel-level correspondence tends to be noisy because of many similar misleading pixels, e.g., backgrounds. To address this issue, in this paper, we propose to explore \textbf{set} \textbf{sim}ilarity (SetSim) for dense self-supervised representation learning. We generalize pixel-wise similarity learning to set-wise one to improve the robustness because sets contain more semantic and structure information. Specifically, by resorting to attentional features of views, we establish corresponding sets, thus filtering out noisy backgrounds that may cause incorrect correspondences. Meanwhile, these attentional features can keep the coherence of the same image across different views to alleviate semantic inconsistency. We further search the cross-view nearest neighbours of sets and employ the structured neighbourhood information to enhance the robustness. Empirical evaluations demonstrate that SetSim is superior to state-of-the-art methods on object detection, keypoint detection, instance segmentation, and semantic segmentation.