Alternating optimization method based on nonnegative matrix factorizations for deep neural networks
This addresses the challenge of avoiding backpropagation's derivative requirements and parameter tuning difficulties for researchers in machine learning, though it appears incremental as it matches rather than surpasses existing performance.
The authors tackled the problem of training deep neural networks without backpropagation by proposing an alternating optimization method based on semi-nonnegative matrix factorizations, achieving error rates similar to conventional backpropagation-based DNNs.
The backpropagation algorithm for calculating gradients has been widely used in computation of weights for deep neural networks (DNNs). This method requires derivatives of objective functions and has some difficulties finding appropriate parameters such as learning rate. In this paper, we propose a novel approach for computing weight matrices of fully-connected DNNs by using two types of semi-nonnegative matrix factorizations (semi-NMFs). In this method, optimization processes are performed by calculating weight matrices alternately, and backpropagation (BP) is not used. We also present a method to calculate stacked autoencoder using a NMF. The output results of the autoencoder are used as pre-training data for DNNs. The experimental results show that our method using three types of NMFs attains similar error rates to the conventional DNNs with BP.