Dynamic Optimization of Neural Network Structures Using Probabilistic Modeling
This addresses the problem of network structure optimization for machine learning practitioners, but it is incremental as it builds on existing probabilistic methods for structure search.
The paper tackles the challenge of selecting and designing appropriate neural network structures by proposing a method that simultaneously optimizes network structure and weight parameters during training using probabilistic modeling. Experimental results on MNIST, CIFAR-10, and CIFAR-100 datasets show that the method finds competitive network structures.
Deep neural networks (DNNs) are powerful machine learning models and have succeeded in various artificial intelligence tasks. Although various architectures and modules for the DNNs have been proposed, selecting and designing the appropriate network structure for a target problem is a challenging task. In this paper, we propose a method to simultaneously optimize the network structure and weight parameters during neural network training. We consider a probability distribution that generates network structures, and optimize the parameters of the distribution instead of directly optimizing the network structure. The proposed method can apply to the various network structure optimization problems under the same framework. We apply the proposed method to several structure optimization problems such as selection of layers, selection of unit types, and selection of connections using the MNIST, CIFAR-10, and CIFAR-100 datasets. The experimental results show that the proposed method can find the appropriate and competitive network structures.