Spectral Architecture Search for Neural Network Models
This addresses the problem of efficient neural network architecture search for researchers and practitioners, but it appears incremental as it builds on existing search methods with a spectral twist.
The paper tackles the challenge of architecture design and optimization in neural networks by introducing SPARCS, a spectral-based architecture search protocol that uses inter-layer transfer matrices to explore architectures via continuous manifolds, enabling gradient-based optimization; results show it yields self-emerging architectures with minimal expressivity and reduced parameter counts on benchmark models.
Architecture design and optimization are challenging problems in the field of artificial neural networks. Working in this context, we here present SPARCS (SPectral ARchiteCture Search), a novel architecture search protocol which exploits the spectral attributes of the inter-layer transfer matrices. SPARCS allows one to explore the space of possible architectures by spanning continuous and differentiable manifolds, thus enabling for gradient-based optimization algorithms to be eventually employed. With reference to simple benchmark models, we show that the newly proposed method yields a self-emerging architecture with a minimal degree of expressivity to handle the task under investigation and with a reduced parameter count as compared to other viable alternatives.