Can Cross Encoders Produce Useful Sentence Embeddings?
This addresses the efficiency and accuracy trade-offs in sentence embedding models for information retrieval, offering a novel approach to leverage CEs beyond re-ranking.
The paper tackles the problem of using cross encoders (CEs) for sentence embeddings, showing that embeddings from earlier CE layers can be integrated into information retrieval pipelines, and demonstrates a method to distill a lighter-weight dual encoder (DE) with a 5.15x speedup in inference time.
Cross encoders (CEs) are trained with sentence pairs to detect relatedness. As CEs require sentence pairs at inference, the prevailing view is that they can only be used as re-rankers in information retrieval pipelines. Dual encoders (DEs) are instead used to embed sentences, where sentence pairs are encoded by two separate encoders with shared weights at training, and a loss function that ensures the pair's embeddings lie close in vector space if the sentences are related. DEs however, require much larger datasets to train, and are less accurate than CEs. We report a curious finding that embeddings from earlier layers of CEs can in fact be used within an information retrieval pipeline. We show how to exploit CEs to distill a lighter-weight DE, with a 5.15x speedup in inference time.