Idea2Plan: Exploring AI-Powered Research Planning
This work addresses the need for systematic evaluation of LLMs' research planning capabilities, which is crucial for scientists and autonomous research agents, though it is incremental as it builds on existing LLM applications.
The paper tackles the problem of measuring how well large language models (LLMs) can convert research ideas into structured plans, introducing the Idea2Plan task and benchmark based on 200 ICML 2025 papers, and finds that GPT-5 and GPT-5-mini perform best but with room for improvement.
Large language models (LLMs) have demonstrated significant potential to accelerate scientific discovery as valuable tools for analyzing data, generating hypotheses, and supporting innovative approaches in various scientific fields. In this work, we investigate how LLMs can handle the transition from conceptual research ideas to well-structured research plans. Effective research planning not only supports scientists in advancing their research but also represents a crucial capability for the development of autonomous research agents. Despite its importance, the field lacks a systematic understanding of LLMs' research planning capability. To rigorously measure this capability, we introduce the Idea2Plan task and Idea2Plan Bench, a benchmark built from 200 ICML 2025 Spotlight and Oral papers released after major LLM training cutoffs. Each benchmark instance includes a research idea and a grading rubric capturing the key components of valid plans. We further propose Idea2Plan JudgeEval, a complementary benchmark to assess the reliability of LLM-based judges against expert annotations. Experimental results show that GPT-5 and GPT-5-mini achieve the strongest performance on the benchmark, though substantial headroom remains for future improvement. Our study provides new insights into LLMs' capability for research planning and lay the groundwork for future progress.