Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedback
This work addresses the problem of enhancing automation in scientific research for researchers, though it appears incremental as it builds on existing AI-assisted methods.
The authors tackled the challenge of automating scientific research by introducing Dolphin, a closed-loop LLM-driven framework that generates novel ideas, implements them with refined code, and uses feedback to improve performance iteratively, achieving results comparable to state-of-the-art in tasks like 3D point classification.
The scientific research paradigm is undergoing a profound transformation owing to the development of Artificial Intelligence (AI). Recent works demonstrate that various AI-assisted research methods can largely improve research efficiency by improving data analysis, accelerating computation, and fostering novel idea generation. To further move towards the ultimate goal (i.e., automatic scientific research), in this paper, we introduce Dolphin, a closed-loop LLM-driven framework to enhance the automation level of scientific research. Dolphin first generates novel ideas based on feedback from previous experiments and relevant papers ranked by the topic and task attributes. Then, the generated ideas can be implemented using a code template refined and debugged with the designed exception-traceback-guided local code structure. Finally, Dolphin automatically analyzes the results of each idea and feeds the results back to the next round of idea generation. Experiments are conducted on the benchmark datasets of different topics and a subset of MLE-bench. Results show that Dolphin can continuously improve the performance of the input topic in a loop. We highlight that Dolphin can automatically propose methods that are comparable to the state-of-the-art in some tasks such as 3D point classification.