CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation
This addresses the problem of constrained design spaces and limited evaluation in autonomous scientific discovery for researchers, representing a qualitative shift but still incremental in novelty.
The paper tackles the limitations of current autonomous scientific discovery systems by introducing CodeScientist, which uses genetic search over research articles and codeblocks to conduct hundreds of automated experiments, resulting in 19 discoveries with 6 judged as sound and novel.
Despite the surge of interest in autonomous scientific discovery (ASD) of software artifacts (e.g., improved ML algorithms), current ASD systems face two key limitations: (1) they largely explore variants of existing codebases or similarly constrained design spaces, and (2) they produce large volumes of research artifacts (such as automatically generated papers and code) that are typically evaluated using conference-style paper review with limited evaluation of code. In this work we introduce CodeScientist, a novel ASD system that frames ideation and experiment construction as a form of genetic search jointly over combinations of research articles and codeblocks defining common actions in a domain (like prompting a language model). We use this paradigm to conduct hundreds of automated experiments on machine-generated ideas broadly in the domain of agents and virtual environments, with the system returning 19 discoveries, 6 of which were judged as being both at least minimally sound and incrementally novel after a multi-faceted evaluation beyond that typically conducted in prior work, including external (conference-style) review, code review, and replication attempts. Moreover, the discoveries span new tasks, agents, metrics, and data, suggesting a qualitative shift from benchmark optimization to broader discoveries.