Condenser: a Pre-training Architecture for Dense Retrieval
This work addresses the challenge of training dense encoders efficiently, particularly in low-data scenarios, for researchers and practitioners in information retrieval and natural language processing, representing a novel architectural advancement rather than an incremental improvement.
The paper tackles the problem of dense encoders requiring extensive data and sophisticated training by identifying that standard language models' attention structures are not suitable for dense representation aggregation. It introduces Condenser, a pre-training architecture that conditions language model prediction on dense representation, achieving significant improvements over standard models on text retrieval and similarity tasks.
Pre-trained Transformer language models (LM) have become go-to text representation encoders. Prior research fine-tunes deep LMs to encode text sequences such as sentences and passages into single dense vector representations for efficient text comparison and retrieval. However, dense encoders require a lot of data and sophisticated techniques to effectively train and suffer in low data situations. This paper finds a key reason is that standard LMs' internal attention structure is not ready-to-use for dense encoders, which needs to aggregate text information into the dense representation. We propose to pre-train towards dense encoder with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. Our experiments show Condenser improves over standard LM by large margins on various text retrieval and similarity tasks.