Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning
This work addresses efficiency and performance in information retrieval for search systems, but it is incremental as it builds on existing BERT-based models.
The paper tackles improving BERT-based bi-encoder neural ranking models by introducing lightweight fine-tuning methods (adapter-based, prompt-based, hybrid) and semi-Siamese architectures, achieving performance gains across multiple datasets like Robust04, ClueWeb09b, and MS-MARCO.
A BERT-based Neural Ranking Model (NRM) can be either a crossencoder or a bi-encoder. Between the two, bi-encoder is highly efficient because all the documents can be pre-processed before the actual query time. In this work, we show two approaches for improving the performance of BERT-based bi-encoders. The first approach is to replace the full fine-tuning step with a lightweight fine-tuning. We examine lightweight fine-tuning methods that are adapter-based, prompt-based, and hybrid of the two. The second approach is to develop semi-Siamese models where queries and documents are handled with a limited amount of difference. The limited difference is realized by learning two lightweight fine-tuning modules, where the main language model of BERT is kept common for both query and document. We provide extensive experiment results for monoBERT, TwinBERT, and ColBERT where three performance metrics are evaluated over Robust04, ClueWeb09b, and MS-MARCO datasets. The results confirm that both lightweight fine-tuning and semi-Siamese are considerably helpful for improving BERT-based bi-encoders. In fact, lightweight fine-tuning is helpful for crossencoder, too