Minimum number of neurons in fully connected layers of a given neural network (the first approximation)
This addresses the challenge of optimizing network architecture efficiently for researchers and practitioners, though it is an incremental improvement with limitations in guarantees and solution scope.
The paper tackles the problem of determining the minimum number of neurons in fully connected layers of neural networks without retraining, proposing an algorithm based on cross-validation and truncated singular value decomposition autoencoders. It shows that this minimum is an internal property of the solution, tested on classification and regression datasets.
This paper presents an algorithm for searching for the minimum number of neurons in fully connected layers of an arbitrary network solving given problem, which does not require multiple training of the network with different number of neurons. The algorithm is based at training the initial wide network using the cross-validation method over at least two folds. Then by using truncated singular value decomposition autoencoder inserted after the studied layer of trained network we search the minimum number of neurons in inference only mode of the network. It is shown that the minimum number of neurons in a fully connected layer could be interpreted not as network hyperparameter associated with the other hyperparameters of the network, but as internal (latent) property of the solution, determined by the network architecture, the training dataset, layer position, and the quality metric used. So the minimum number of neurons can be estimated for each hidden fully connected layer independently. The proposed algorithm is the first approximation for estimating the minimum number of neurons in the layer, since, on the one hand, the algorithm does not guarantee that a neural network with the found number of neurons can be trained to the required quality, and on the other hand, it searches for the minimum number of neurons in a limited class of possible solutions. The solution was tested on several datasets in classification and regression problems.