Learning Retrieval Models with Sparse Autoencoders
This work addresses the need for efficient and effective retrieval models in multilingual and out-of-domain settings, representing an incremental improvement over existing LSR methods.
The paper tackled the problem of improving learned sparse retrieval (LSR) by using sparse autoencoders (SAEs) to create more semantically structured and language-agnostic representations, resulting in SPLARE-7B achieving top results on MMTEB's multilingual and English retrieval tasks.
Sparse autoencoders (SAEs) provide a powerful mechanism for decomposing the dense representations produced by Large Language Models (LLMs) into interpretable latent features. We posit that SAEs constitute a natural foundation for Learned Sparse Retrieval (LSR), whose objective is to encode queries and documents into high-dimensional sparse representations optimized for efficient retrieval. In contrast to existing LSR approaches that project input sequences into the vocabulary space, SAE-based representations offer the potential to produce more semantically structured, expressive, and language-agnostic features. Building on this insight, we introduce SPLARE, a method to train SAE-based LSR models. Our experiments, relying on recently released open-source SAEs, demonstrate that this technique consistently outperforms vocabulary-based LSR in multilingual and out-of-domain settings. SPLARE-7B, a multilingual retrieval model capable of producing generalizable sparse latent embeddings for a wide range of languages and domains, achieves top results on MMTEB's multilingual and English retrieval tasks. We also developed a 2B-parameter variant with a significantly lighter footprint.