Transfer NAS: Knowledge Transfer between Search Spaces with Transformer Agents
This addresses a bottleneck in NAS for researchers and practitioners by reducing computational costs and improving efficiency in architecture search.
The paper tackles the problem of restarting learning from scratch for different Neural Architecture Search (NAS) search spaces by proposing a Transformer-based agent that enables joint training and efficient knowledge transfer between multiple search spaces and tasks.
Recent advances in Neural Architecture Search (NAS) have produced state-of-the-art architectures on several tasks. NAS shifts the efforts of human experts from developing novel architectures directly to designing architecture search spaces and methods to explore them efficiently. The search space definition captures prior knowledge about the properties of the architectures and it is crucial for the complexity and the performance of the search algorithm. However, different search space definitions require restarting the learning process from scratch. We propose a novel agent based on the Transformer that supports joint training and efficient transfer of prior knowledge between multiple search spaces and tasks.