Automatic Question-Answer Generation for Long-Tail Knowledge
This addresses the scarcity of QA datasets for tail entities, which is a bottleneck for improving LLM accuracy in open-domain question answering, though it is incremental as it builds on existing methods.
The paper tackles the problem of LLMs struggling with long-tail knowledge by proposing an automatic method to generate specialized QA datasets for tail entities, and finds that using external resources like Wikipedia and Wikidata improves performance.
Pretrained Large Language Models (LLMs) have gained significant attention for addressing open-domain Question Answering (QA). While they exhibit high accuracy in answering questions related to common knowledge, LLMs encounter difficulties in learning about uncommon long-tail knowledge (tail entities). Since manually constructing QA datasets demands substantial human resources, the types of existing QA datasets are limited, leaving us with a scarcity of datasets to study the performance of LLMs on tail entities. In this paper, we propose an automatic approach to generate specialized QA datasets for tail entities and present the associated research challenges. We conduct extensive experiments by employing pretrained LLMs on our newly generated long-tail QA datasets, comparing their performance with and without external resources including Wikipedia and Wikidata knowledge graphs.