STAR: Synthesis of Tailored Architectures
This work addresses the problem of expensive and limited architecture optimization for researchers and practitioners, offering a novel approach that is incremental in building on existing methods.
The paper tackles the challenge of optimizing deep learning architectures by proposing STAR, a method that synthesizes tailored architectures using a novel search space and evolutionary algorithms, achieving improvements over optimized Transformers and hybrid models in autoregressive language modeling.
Iterative improvement of model architectures is fundamental to deep learning: Transformers first enabled scaling, and recent advances in model hybridization have pushed the quality-efficiency frontier. However, optimizing architectures remains challenging and expensive. Current automated or manual approaches fall short, largely due to limited progress in the design of search spaces and due to the simplicity of resulting patterns and heuristics. In this work, we propose a new approach for the synthesis of tailored architectures (STAR). Our approach combines a novel search space based on the theory of linear input-varying systems, supporting a hierarchical numerical encoding into architecture genomes. STAR genomes are automatically refined and recombined with gradient-free, evolutionary algorithms to optimize for multiple model quality and efficiency metrics. Using STAR, we optimize large populations of new architectures, leveraging diverse computational units and interconnection patterns, improving over highly-optimized Transformers and striped hybrid models on the frontier of quality, parameter size, and inference cache for autoregressive language modeling.