ColBERTSaR: Sparsified ColBERT Index via Product Quantization
For practitioners deploying ColBERT-based retrieval at scale, this work reduces storage and query-time overhead, but the approach is incremental (quantization of existing embeddings).
ColBERT's index is large and slow due to gathering/decompression. The authors propose embedding quantization to create a true inverted index, reducing index size by 50-70% vs. one-bit PLAID while maintaining retrieval effectiveness.
While ColBERT is an effective neural retrieval architecture, it requires a heavy index structure to support candidate set retrieval based on approximated token embeddings, gathering and decompressing document token embeddings, and applying the MaxSim operation. Indexes in PLAID and similar ColBERT implementations require five to ten times the disk storage of the original raw text, which limits their scalability. Furthermore, prior work has identified that the gathering and decompression stages are the primary inefficiencies at query time. Limiting the number of document tokens that must be gathered by thresholding and score approximation does not eliminate the need for the entire index to support ad hoc queries. In this work, we propose an embedding quantization approach that turns a ColBERT index into a true inverted index. We show that, theoretically, ColBERT with embedding quantization is equivalent to learned-sparse retrieval except for the scoring mechanism. Empirically, we demonstrate that our index is 50-70% smaller than a one-bit PLAID index while retaining retrieval effectiveness.