League: Leaderboard Generation on Demand
This provides an incremental solution for researchers in rapidly evolving fields like AI to efficiently track and compare papers, but it is domain-specific to leaderboard generation.
The paper tackles the problem of automatically generating leaderboards for AI research topics by introducing the Leaderboard Auto Generation (LAG) framework, which addresses challenges like multi-document summarization and fair experiment comparison, with results showing high-quality leaderboards.
This paper introduces Leaderboard Auto Generation (LAG), a novel and well-organized framework for automatic generation of leaderboards on a given research topic in rapidly evolving fields like Artificial Intelligence (AI). Faced with a large number of AI papers updated daily, it becomes difficult for researchers to track every paper's proposed methods, experimental results, and settings, prompting the need for efficient automatic leaderboard construction. While large language models (LLMs) offer promise in automating this process, challenges such as multi-document summarization, leaderboard generation, and experiment fair comparison still remain under exploration. LAG solves these challenges through a systematic approach that involves the paper collection, experiment results extraction and integration, leaderboard generation, and quality evaluation. Our contributions include a comprehensive solution to the leaderboard construction problem, a reliable evaluation method, and experimental results showing the high quality of leaderboards.