FAT ALBERT: Finding Answers in Large Texts using Semantic Similarity Attention Layer based on BERT
This work addresses the challenge of scaling BERT for large-text question answering, providing an incremental improvement for NLP applications like movie-based QA systems.
The paper tackles the problem of multiple-choice question answering on large text corpora by developing a BERT-based model enhanced with a semantic similarity attention layer to extract high-influence sentences, achieving state-of-the-art results with 87.79% test accuracy on the MovieQA challenge.
Machine based text comprehension has always been a significant research field in natural language processing. Once a full understanding of the text context and semantics is achieved, a deep learning model can be trained to solve a large subset of tasks, e.g. text summarization, classification and question answering. In this paper we focus on the question answering problem, specifically the multiple choice type of questions. We develop a model based on BERT, a state-of-the-art transformer network. Moreover, we alleviate the ability of BERT to support large text corpus by extracting the highest influence sentences through a semantic similarity model. Evaluations of our proposed model demonstrate that it outperforms the leading models in the MovieQA challenge and we are currently ranked first in the leader board with test accuracy of 87.79%. Finally, we discuss the model shortcomings and suggest possible improvements to overcome these limitations.