Context-Aware Transformer Pre-Training for Answer Sentence Selection
This work addresses the need for more accurate AS2 models in Question Answering pipelines, though it is incremental as it builds on existing pre-trained transformers.
The paper tackled the problem of improving Answer Sentence Selection (AS2) by proposing three pre-training objectives that mimic the downstream fine-tuning task, resulting in up to 8% accuracy improvement on some datasets.
Answer Sentence Selection (AS2) is a core component for building an accurate Question Answering pipeline. AS2 models rank a set of candidate sentences based on how likely they answer a given question. The state of the art in AS2 exploits pre-trained transformers by transferring them on large annotated datasets, while using local contextual information around the candidate sentence. In this paper, we propose three pre-training objectives designed to mimic the downstream fine-tuning task of contextual AS2. This allows for specializing LMs when fine-tuning for contextual AS2. Our experiments on three public and two large-scale industrial datasets show that our pre-training approaches (applied to RoBERTa and ELECTRA) can improve baseline contextual AS2 accuracy by up to 8% on some datasets.