Non-Gradient Manifold Neural Network
This work addresses convergence speed and classification accuracy issues in deep learning, offering a novel optimization approach that could benefit researchers and practitioners in machine learning.
The authors tackled slow convergence in deep neural networks and the loss of data distribution information in classification by proposing a non-gradient manifold neural network with closed-form solutions, achieving rapid convergence and superior performance in experiments.
Deep neural network (DNN) generally takes thousands of iterations to optimize via gradient descent and thus has a slow convergence. In addition, softmax, as a decision layer, may ignore the distribution information of the data during classification. Aiming to tackle the referred problems, we propose a novel manifold neural network based on non-gradient optimization, i.e., the closed-form solutions. Considering that the activation function is generally invertible, we reconstruct the network via forward ridge regression and low rank backward approximation, which achieve the rapid convergence. Moreover, by unifying the flexible Stiefel manifold and adaptive support vector machine, we devise the novel decision layer which efficiently fits the manifold structure of the data and label information. Consequently, a jointly non-gradient optimization method is designed to generate the network with closed-form results. Eventually, extensive experiments validate the superior performance of the model.