Patch-Level Kernel Alignment for Dense Self-Supervised Learning
This work addresses the problem of enhancing fine-grained semantic understanding in vision models for computer vision researchers, offering an efficient and flexible solution that is incremental over prior dense SSL methods.
The paper tackles the limitations of existing dense self-supervised learning methods by introducing Patch-level Kernel Alignment (PaKA), a non-parametric, kernel-based approach that improves dense representations with lightweight post-training, achieving state-of-the-art performance across dense vision benchmarks in only 14 hours on a single GPU.
Dense self-supervised learning (SSL) methods showed its effectiveness in enhancing the fine-grained semantic understandings of vision models. However, existing approaches often rely on parametric assumptions or complex post-processing (e.g., clustering, sorting), limiting their flexibility and stability. To overcome these limitations, we introduce Patch-level Kernel Alignment (PaKA), a non-parametric, kernel-based approach that improves the dense representations of pretrained vision encoders with a post-(pre)training. Our method propose a robust and effective alignment objective that captures statistical dependencies which matches the intrinsic structure of high-dimensional dense feature distributions. In addition, we revisit the augmentation strategies inherited from image-level SSL and propose a refined augmentation strategy for dense SSL. Our framework improves dense representations by conducting a lightweight post-training stage on top of a pretrained model. With only 14 hours of additional training on a single GPU, our method achieves state-of-the-art performance across a range of dense vision benchmarks, demonstrating both efficiency and effectiveness.