Improving Research Idea Generation Through Data: An Empirical Investigation in Social Science
This work addresses the challenge of generating practical research ideas for social science researchers, though it is incremental as it builds on existing LLM methods with data augmentation.
The paper tackled the problem of generating feasible and effective research ideas using large language models (LLMs) by augmenting them with relevant data, resulting in a 20% improvement in feasibility and a 7% improvement in overall quality of selected ideas in social science experiments.
Recent advancements in large language models (LLMs) have shown promise in generating novel research ideas. However, these ideas often face challenges related to feasibility and expected effectiveness. This paper explores how augmenting LLMs with relevant data during the idea generation process can enhance the quality of generated ideas. We introduce two ways of incorporating data: (1) providing metadata during the idea generation stage to guide LLMs toward feasible directions, and (2) adding automatic validation during the idea selection stage to assess the empirical plausibility of hypotheses within ideas. We conduct experiments in the social science domain, specifically with climate negotiation topics, and find that metadata improves the feasibility of generated ideas by 20%, while automatic validation improves the overall quality of selected ideas by 7%. A human study shows that LLM-generated ideas, along with their related data and validation processes, inspire researchers to propose research ideas with higher quality. Our work highlights the potential of data-driven research idea generation, and underscores the practical utility of LLM-assisted ideation in real-world academic settings.