Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval
This addresses interpretability and efficiency issues in dense retrieval for AI and information retrieval communities, though it is incremental as it builds on existing DPR and SAE methods.
The paper tackles the lack of interpretability in Dense Passage Retrieval (DPR) models by using Sparse Autoencoders (SAEs) to decompose dense embeddings into interpretable latent concepts with natural language descriptions, and introduces Concept-Level Sparse Retrieval (CL-SR) that achieves high efficiency while maintaining robust performance.
Despite their strong performance, Dense Passage Retrieval (DPR) models suffer from a lack of interpretability. In this work, we propose a novel interpretability framework that leverages Sparse Autoencoders (SAEs) to decompose previously uninterpretable dense embeddings from DPR models into distinct, interpretable latent concepts. We generate natural language descriptions for each latent concept, enabling human interpretations of both the dense embeddings and the query-document similarity scores of DPR models. We further introduce Concept-Level Sparse Retrieval (CL-SR), a retrieval framework that directly utilizes the extracted latent concepts as indexing units. CL-SR effectively combines the semantic expressiveness of dense embeddings with the transparency and efficiency of sparse representations. We show that CL-SR achieves high index-space and computational efficiency while maintaining robust performance across vocabulary and semantic mismatches.