LEGOBench: Scientific Leaderboard Generation Benchmark
This addresses the problem of information overload for researchers by providing a benchmark for leaderboard generation, though it is incremental as it builds on existing data and models.
The authors tackled the challenge of staying informed about state-of-the-art research by introducing LEGOBench, a benchmark for evaluating systems that generate scientific leaderboards, using data from arXiv and PapersWithCode, and found that state-of-the-art models show significant performance gaps in this task.
The ever-increasing volume of paper submissions makes it difficult to stay informed about the latest state-of-the-art research. To address this challenge, we introduce LEGOBench, a benchmark for evaluating systems that generate scientific leaderboards. LEGOBench is curated from 22 years of preprint submission data on arXiv and more than 11k machine learning leaderboards on the PapersWithCode portal. We present four graph-based and two language model-based leaderboard generation task configurations. We evaluate popular encoder-only scientific language models as well as decoder-only large language models across these task configurations. State-of-the-art models showcase significant performance gaps in automatic leaderboard generation on LEGOBench. The code is available on GitHub ( https://github.com/lingo-iitgn/LEGOBench ) and the dataset is hosted on OSF ( https://osf.io/9v2py/?view_only=6f91b0b510df498ba01595f8f278f94c ).