Contrastive Neighborhood Alignment
This addresses the challenge of preserving local structure in unsupervised learning for tasks such as model distillation and legacy updates, though it appears incremental as it builds on existing contrastive and manifold learning techniques.
The paper tackles the problem of maintaining the topology of learned features across models by introducing Contrastive Neighborhood Alignment (CNA), which uses a contrastive loss to ensure that data points mapped to nearby representations by a source model are similarly mapped by a target model, and experiments show it improves performance compared to competing methods in domains like manifold learning and model distillation.
We present Contrastive Neighborhood Alignment (CNA), a manifold learning approach to maintain the topology of learned features whereby data points that are mapped to nearby representations by the source (teacher) model are also mapped to neighbors by the target (student) model. The target model aims to mimic the local structure of the source representation space using a contrastive loss. CNA is an unsupervised learning algorithm that does not require ground-truth labels for the individual samples. CNA is illustrated in three scenarios: manifold learning, where the model maintains the local topology of the original data in a dimension-reduced space; model distillation, where a small student model is trained to mimic a larger teacher; and legacy model update, where an older model is replaced by a more powerful one. Experiments show that CNA is able to capture the manifold in a high-dimensional space and improves performance compared to the competing methods in their domains.