Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
This work addresses the need for efficient and high-fidelity adaptive embeddings in real-world applications like retrieval and search, offering a novel alternative to existing methods.
The paper tackles the problem of adaptive representation learning for large-scale systems by proposing Contrastive Sparse Representation (CSR), which sparsifies pre-trained embeddings to achieve flexible inference with minimal overhead, outperforming Matryoshka Representation Learning in accuracy and retrieval speed while reducing training time significantly.
Many large-scale systems rely on high-quality deep representations (embeddings) to facilitate tasks like retrieval, search, and generative modeling. Matryoshka Representation Learning (MRL) recently emerged as a solution for adaptive embedding lengths, but it requires full model retraining and suffers from noticeable performance degradations at short lengths. In this paper, we show that sparse coding offers a compelling alternative for achieving adaptive representation with minimal overhead and higher fidelity. We propose Contrastive Sparse Representation (CSR), a method that sparsifies pre-trained embeddings into a high-dimensional but selectively activated feature space. By leveraging lightweight autoencoding and task-aware contrastive objectives, CSR preserves semantic quality while allowing flexible, cost-effective inference at different sparsity levels. Extensive experiments on image, text, and multimodal benchmarks demonstrate that CSR consistently outperforms MRL in terms of both accuracy and retrieval speed-often by large margins-while also cutting training time to a fraction of that required by MRL. Our results establish sparse coding as a powerful paradigm for adaptive representation learning in real-world applications where efficiency and fidelity are both paramount. Code is available at https://github.com/neilwen987/CSR_Adaptive_Rep