Effective Context Selection in LLM-based Leaderboard Generation: An Empirical Study
This work addresses the challenge of reliable and efficient leaderboard generation for AI researchers, though it is incremental in nature.
This paper tackled the problem of generating AI research leaderboards by extracting quadruples from scholarly articles using LLMs, and found that effective context selection improves accuracy and reduces hallucinations.
This paper explores the impact of context selection on the efficiency of Large Language Models (LLMs) in generating Artificial Intelligence (AI) research leaderboards, a task defined as the extraction of (Task, Dataset, Metric, Score) quadruples from scholarly articles. By framing this challenge as a text generation objective and employing instruction finetuning with the FLAN-T5 collection, we introduce a novel method that surpasses traditional Natural Language Inference (NLI) approaches in adapting to new developments without a predefined taxonomy. Through experimentation with three distinct context types of varying selectivity and length, our study demonstrates the importance of effective context selection in enhancing LLM accuracy and reducing hallucinations, providing a new pathway for the reliable and efficient generation of AI leaderboards. This contribution not only advances the state of the art in leaderboard generation but also sheds light on strategies to mitigate common challenges in LLM-based information extraction.