Growing an architecture for a neural network
This work addresses the challenge of designing efficient neural network architectures for researchers and practitioners, offering a novel approach that could improve performance in image-related tasks, though it appears incremental in the context of architecture search methods.
The authors tackled the problem of automatic neural network architecture search by proposing an algorithm that alternates pruning connections and adding neurons, allowing for arbitrary graph structures without layered constraints, and demonstrated it on brightness prediction and bivariate function approximation tasks, where it significantly outperformed standard solutions.
We propose a new kind of automatic architecture search algorithm. The algorithm alternates pruning connections and adding neurons, and it is not restricted to layered architectures only. Here architecture is an arbitrary oriented graph with some weights (along with some biases and an activation function), so there may be no layered structure in such a network. The algorithm minimizes the complexity of staying within a given error. We demonstrate our algorithm on the brightness prediction problem of the next point through the previous points on an image. Our second test problem is the approximation of the bivariate function defining the brightness of a black and white image. Our optimized networks significantly outperform the standard solution for neural network architectures in both cases.