TART: Token-based Architecture Transformer for Neural Network Performance Prediction
This work addresses the challenge of reducing manual expertise in neural architecture search for researchers, though it is incremental as it builds on existing transformer methods for performance prediction.
The paper tackles the problem of automating neural architecture design by proposing TART, a token-based architecture transformer that predicts neural network performance without training candidate networks, achieving state-of-the-art results on the DeepNets-1M dataset.
In the realm of neural architecture design, achieving high performance is largely reliant on the manual expertise of researchers. Despite the emergence of Neural Architecture Search (NAS) as a promising technique for automating this process, current NAS methods still require human input to expand the search space and cannot generate new architectures. This paper explores the potential of Transformers in comprehending neural architectures and their performance, with the objective of establishing the foundation for utilizing Transformers to generate novel networks. We propose the Token-based Architecture Transformer (TART), which predicts neural network performance without the need to train candidate networks. TART attains state-of-the-art performance on the DeepNets-1M dataset for performance prediction tasks without edge information, indicating the potential of Transformers to aid in discovering novel and high-performing neural architectures.