CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents
This addresses the problem of limited code generation capabilities for scientific coding agents, offering an automated solution to enhance ASD systems without manual effort, though it is incremental as it builds on existing ASD frameworks.
The paper tackles the limitation of Automated Scientific Discovery (ASD) systems in generating reliable code by introducing CodeDistiller, which automatically distills GitHub repositories into a library of domain-specific code examples, resulting in 74% functional examples and improved experiment accuracy and completeness in downstream evaluations.
Automated Scientific Discovery (ASD) systems can help automatically generate and run code-based experiments, but their capabilities are limited by the code they can reliably generate from parametric knowledge alone. As a result, current systems either mutate a small number of manually-crafted experiment examples, or operate solely from parametric knowledge, limiting quality and reach. We introduce CodeDistiller, a system that automatically distills large collections of scientific Github repositories into a vetted library of working domain-specific code examples, allowing ASD agents to expand their capabilities without manual effort. Using a combination of automatic and domain-expert evaluation on 250 materials science repositories, we find the best model is capable of producing functional examples for 74% of repositories, while our downstream evaluation shows an ASD agent augmented with a CodeDistiller generated library produces more accurate, complete, and scientifically sound experiments than an agent with only general materials-science code examples.