Optimizing Deep Neural Networks through Neuroevolution with Stochastic Gradient Descent
This work addresses the challenge of optimizing deep neural networks for improved performance, which is a problem for researchers and practitioners in machine learning.
This paper proposes a novel approach that combines neuroevolution with stochastic gradient descent (SGD) to optimize deep neural networks (DNNs). The method, which also includes a hierarchical cluster-based suppression algorithm, consistently outperforms DNNs optimized by SGD alone and state-of-the-art deep networks across four representative DNNs and datasets.
Deep neural networks (DNNs) have achieved remarkable success in computer vision; however, training DNNs for satisfactory performance remains challenging and suffers from sensitivity to empirical selections of an optimization algorithm for training. Stochastic gradient descent (SGD) is dominant in training a DNN by adjusting neural network weights to minimize the DNNs loss function. As an alternative approach, neuroevolution is more in line with an evolutionary process and provides some key capabilities that are often unavailable in SGD, such as the heuristic black-box search strategy based on individual collaboration in neuroevolution. This paper proposes a novel approach that combines the merits of both neuroevolution and SGD, enabling evolutionary search, parallel exploration, and an effective probe for optimal DNNs. A hierarchical cluster-based suppression algorithm is also developed to overcome similar weight updates among individuals for improving population diversity. We implement the proposed approach in four representative DNNs based on four publicly-available datasets. Experiment results demonstrate that the four DNNs optimized by the proposed approach all outperform corresponding ones optimized by only SGD on all datasets. The performance of DNNs optimized by the proposed approach also outperforms state-of-the-art deep networks. This work also presents a meaningful attempt for pursuing artificial general intelligence.