MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?
This addresses the need for rigorous, objective evaluation of AI research capabilities in the ML community, though it is incremental as it builds on prior work like AI Scientist.
The authors tackled the problem of evaluating language agents' ability to solve machine learning research challenges by introducing MLRC-Bench, a benchmark that measures proposing and implementing novel methods, and found that even the best agent closed only 9.3% of the gap between baseline and top human scores.
We introduce MLRC-Bench, a benchmark designed to quantify how effectively language agents can tackle challenging Machine Learning (ML) Research Competitions, with a focus on open research problems that demand novel methodologies. Unlike prior work, e.g., AI Scientist, which evaluates the end-to-end agentic pipeline by using LLM-as-a-judge, MLRC-Bench measures the key steps of proposing and implementing novel research methods and evaluates them with rigorous protocol and objective metrics. Our curated suite of 7 competition tasks reveals significant challenges for LLM agents. Even the best-performing tested agent (gemini-exp-1206 under MLAB) closes only 9.3% of the gap between baseline and top human participant scores. Furthermore, our analysis reveals a misalignment between the LLM-judged innovation and actual performance on cutting-edge ML research problems. MLRC-Bench is a dynamic benchmark, designed to grow with new ML competitions and encourage rigorous, objective evaluations of AI research capabilities. Our leaderboard and code are available at: https://huggingface.co/spaces/launch/MLRC_Bench