Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection
This work addresses the need for more efficient answer sentence selection in QA systems, offering a method to reduce reliance on large labeled datasets, though it is incremental in nature.
The paper tackled the problem of answer sentence selection in QA systems by proposing three novel sentence-level transformer pre-training objectives that incorporate paragraph-level semantics, resulting in improved performance over baseline models like RoBERTa and ELECTRA on multiple datasets.
An important task for designing QA systems is answer sentence selection (AS2): selecting the sentence containing (or constituting) the answer to a question from a set of retrieved relevant documents. In this paper, we propose three novel sentence-level transformer pre-training objectives that incorporate paragraph-level semantics within and across documents, to improve the performance of transformers for AS2, and mitigate the requirement of large labeled datasets. Specifically, the model is tasked to predict whether: (i) two sentences are extracted from the same paragraph, (ii) a given sentence is extracted from a given paragraph, and (iii) two paragraphs are extracted from the same document. Our experiments on three public and one industrial AS2 datasets demonstrate the empirical superiority of our pre-trained transformers over baseline models such as RoBERTa and ELECTRA for AS2.