Exploring Dual Encoder Architectures for Question Answering
This work addresses the challenge of improving retrieval efficiency in question answering systems, offering incremental architectural enhancements for researchers and practitioners in NLP and IR.
The paper tackled the problem of optimizing dual encoder architectures for question answering by comparing Siamese and Asymmetric Dual Encoders, finding that SDE performs significantly better than ADE on benchmarks like MS MARCO, and proposed improved ADE versions that achieve competitive or superior performance by sharing parameters in projection layers.
Dual encoders have been used for question-answering (QA) and information retrieval (IR) tasks with good results. Previous research focuses on two major types of dual encoders, Siamese Dual Encoder (SDE), with parameters shared across two encoders, and Asymmetric Dual Encoder (ADE), with two distinctly parameterized encoders. In this work, we explore different ways in which the dual encoder can be structured, and evaluate how these differences can affect their efficacy in terms of QA retrieval tasks. By evaluating on MS MARCO, open domain NQ and the MultiReQA benchmarks, we show that SDE performs significantly better than ADE. We further propose three different improved versions of ADEs by sharing or freezing parts of the architectures between two encoder towers. We find that sharing parameters in projection layers would enable ADEs to perform competitively with or outperform SDEs. We further explore and explain why parameter sharing in projection layer significantly improves the efficacy of the dual encoders, by directly probing the embedding spaces of the two encoder towers with t-SNE algorithm.