Towards Automated Machine Learning Research
This work addresses the challenge of accelerating research progress for ML practitioners by automating component innovation, though it appears incremental in its approach.
The paper tackles the problem of automating incremental machine learning research by proposing a top-down framework that uses Large Language Models to generate novel components, validate feasibility, and evaluate performance against baselines, with results showing improved efficiency in hypothesis generation and evaluation.
This paper explores a top-down approach to automating incremental advances in machine learning research through component-level innovation, facilitated by Large Language Models (LLMs). Our framework systematically generates novel components, validates their feasibility, and evaluates their performance against existing baselines. A key distinction of this approach lies in how these novel components are generated. Unlike traditional AutoML and NAS methods, which often rely on a bottom-up combinatorial search over predefined, hardcoded base components, our method leverages the cross-domain knowledge embedded in LLMs to propose new components that may not be confined to any hard-coded predefined set. By incorporating a reward model to prioritize promising hypotheses, we aim to improve the efficiency of the hypothesis generation and evaluation process. We hope this approach offers a new avenue for exploration and contributes to the ongoing dialogue in the field.