Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes
For researchers seeking to automate the discovery of training recipes, this work demonstrates a fully autonomous closed-loop system that improves upon public baselines without human intervention.
This paper introduces an auto-research loop where specialist agents autonomously propose hypotheses, edit code, run experiments, and use lineage feedback to improve training recipes. Across 1,197 trials, the system achieved a 0.81% reduction in Parameter Golf validation bpb, a 38.7% increase in NanoChat-D12 CORE, and a 4.59% reduction in CIFAR-10 Airbench96 wallclock time.
We study auto research as a closed empirical loop driven by external measurement. Each submitted trial carries a hypothesis, an executable code edit, an evaluator-owned outcome, and feedback that shapes the next proposal. The output is not a generated paper or a single model checkpoint, but an auditable trajectory of proposals, code diffs, experiments, scores, and failure labels. We instantiate this loop with specialist agents that partition recipe surfaces and share measured lineage across trials. The central empirical finding is that lineage feedback lets agents turn evaluator outcomes, including crashes, budget overruns, size failures, and accuracy-gate misses, into later program-level recipe edits rather than one-shot suggestions. Across 1,197 headline-run trials plus 600 Parameter Golf control trials after one-time setup and launch, humans did not choose proposals, edit recipes, override scores, or repair failed trials during the search. In the three headline runs, the same submitted-trial loop reduces Parameter Golf validation bpb by $0.81\%$, raises NanoChat-D12 CORE by $38.7\%$, and reduces CIFAR-10 Airbench96 wallclock by $4.59\%$, with each task measured by its own external evaluator and legality checks. The trace includes a strict architecture-domain audit of 157 headline-run submissions and program rewrites such as a NanoChat attention-kernel path change. Within this scope the loop autonomously writes code, submits experiments, absorbs feedback, applies and combines known techniques inside each environment, and improves public starting recipes.