DP-KB: Data Programming with Knowledge Bases Improves Transformer Fine Tuning for Answer Sentence Selection
This work addresses the challenge of enhancing transformer fine-tuning for QA tasks, particularly in industry settings like Alexa, by improving knowledge utilization without added inference costs, though it is incremental as it builds on existing data-programming methods.
The paper tackles the problem of transformers' limited ability as implicit knowledge bases for knowledge-intensive tasks by proposing a data-programming technique that enriches training data with KB-derived context, resulting in improved performance on answer sentence selection benchmarks, with increases of up to 4.4% p@1 on TrecQA and 2.3% F1 on a proprietary dataset.
While transformers demonstrate impressive performance on many knowledge intensive (KI) tasks, their ability to serve as implicit knowledge bases (KBs) remains limited, as shown on several slot-filling, question-answering (QA), fact verification, and entity-linking tasks. In this paper, we implement an efficient, data-programming technique that enriches training data with KB-derived context and improves transformer utilization of encoded knowledge when fine-tuning for a particular QA task, namely answer sentence selection (AS2). Our method outperforms state of the art transformer approach on WikiQA and TrecQA, two widely studied AS2 benchmarks, increasing by 2.0% p@1, 1.3% MAP, 1.1% MRR, and 4.4% p@1, 0.9% MAP, 2.4% MRR, respectively. To demonstrate our improvements in an industry setting, we additionally evaluate our approach on a proprietary dataset of Alexa QA pairs, and show increase of 2.3% F1 and 2.0% MAP. We additionally find that these improvements remain even when KB context is omitted at inference time, allowing for the use of our models within existing transformer workflows without additional latency or deployment costs.