Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?
This addresses the problem of knowledge retention in language models for researchers in NLP, but it is incremental as it builds on existing work with a new dataset.
The paper investigates whether generative pre-trained language models like BART can serve as knowledge bases for closed-book question answering, finding that BART struggles to recall training facts with high precision and answer questions even when knowledge is retained, with some promising directions identified.
Recent work has investigated the interesting question using pre-trained language models (PLMs) as knowledge bases for answering open questions. However, existing work is limited in using small benchmarks with high test-train overlaps. We construct a new dataset of closed-book QA using SQuAD, and investigate the performance of BART. Experiments show that it is challenging for BART to remember training facts in high precision, and also challenging to answer closed-book questions even if relevant knowledge is retained. Some promising directions are found, including decoupling the knowledge memorizing process and the QA finetune process, forcing the model to recall relevant knowledge when question answering.