Representation Selection via Cross-Model Agreement using Canonical Correlation Analysis
This work addresses efficiency and performance issues in vision pipelines for researchers and practitioners using pretrained models, though it is incremental as it builds on existing dimensionality reduction techniques.
The paper tackles the problem of overcomplete and model-specific image representations from pretrained encoders by proposing a training-free method using canonical correlation analysis to select and reduce dimensions based on cross-model agreement, achieving up to 12.6% accuracy gains while reducing dimensionality by over 75%.
Modern vision pipelines increasingly rely on pretrained image encoders whose representations are reused across tasks and models, yet these representations are often overcomplete and model-specific. We propose a simple, training-free method to improve the efficiency of image representations via a post-hoc canonical correlation analysis (CCA) operator. By leveraging the shared structure between representations produced by two pre-trained image encoders, our method finds linear projections that serve as a principled form of representation selection and dimensionality reduction, retaining shared semantic content while discarding redundant dimensions. Unlike standard dimensionality reduction techniques such as PCA, which operate on a single embedding space, our approach leverages cross-model agreement to guide representation distillation and refinement. The technique allows representations to be reduced by more than 75% in dimensionality with improved downstream performance, or enhanced at fixed dimensionality via post-hoc representation transfer from larger or fine-tuned models. Empirical results on ImageNet-1k, CIFAR-100, MNIST, and additional benchmarks show consistent improvements over both baseline and PCA-projected representations, with accuracy gains of up to 12.6%.