Delaying Interaction Layers in Transformer-based Encoders for Efficient Open Domain Question Answering
This addresses efficiency issues in ODQA for researchers and practitioners, though it is incremental as it builds on existing transformer methods.
The paper tackles the high computational complexity of transformer-based models in Open Domain Question Answering (ODQA) by proposing a change in architecture to delay attention interactions, resulting in competitive performance and significant speedup, with performance improvements in many cases.
Open Domain Question Answering (ODQA) on a large-scale corpus of documents (e.g. Wikipedia) is a key challenge in computer science. Although transformer-based language models such as Bert have shown on SQuAD the ability to surpass humans for extracting answers in small passages of text, they suffer from their high complexity when faced to a much larger search space. The most common way to tackle this problem is to add a preliminary Information Retrieval step to heavily filter the corpus and only keep the relevant passages. In this paper, we propose a more direct and complementary solution which consists in applying a generic change in the architecture of transformer-based models to delay the attention between subparts of the input and allow a more efficient management of computations. The resulting variants are competitive with the original models on the extractive task and allow, on the ODQA setting, a significant speedup and even a performance improvement in many cases.