Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
This work addresses efficiency bottlenecks in dense retrieval for ranking tasks, offering a practical improvement for information retrieval systems, though it is incremental as it builds on existing ColBERT and distillation methods.
The paper tackles the problem of improving the efficiency of dense retrieval models by distilling the knowledge from ColBERT's complex MaxSim operator into a simpler dot product, enabling faster single-step approximate nearest neighbor search. The result is a model that reduces query latency and storage requirements significantly while maintaining competitive effectiveness, approaching the performance of a much slower BERT-based cross-encoder reranker when combined with sparse representations.
We present an approach to ranking with dense representations that applies knowledge distillation to improve the recently proposed late-interaction ColBERT model. Specifically, we distill the knowledge from ColBERT's expressive MaxSim operator for computing relevance scores into a simple dot product, thus enabling single-step ANN search. Our key insight is that during distillation, tight coupling between the teacher model and the student model enables more flexible distillation strategies and yields better learned representations. We empirically show that our approach improves query latency and greatly reduces the onerous storage requirements of ColBERT, while only making modest sacrifices in terms of effectiveness. By combining our dense representations with sparse representations derived from document expansion, we are able to approach the effectiveness of a standard cross-encoder reranker using BERT that is orders of magnitude slower.