Analytic Network Learning
This addresses the challenge of gradient-based optimization in neural networks for researchers, but it appears incremental as it builds on existing linear matrix equation methods.
The paper tackles the problem of learning weight matrices in multilayer networks by deriving an analytic formulation that avoids gradient computation, enabling layer-by-layer learning after initialization. Experiments on synthetic and real-world datasets validate its numerical feasibility, though no concrete performance numbers are provided.
Based on the property that solving the system of linear matrix equations via the column space and the row space projections boils down to an approximation in the least squares error sense, a formulation for learning the weight matrices of the multilayer network can be derived. By exploiting into the vast number of feasible solutions of these interdependent weight matrices, the learning can be performed analytically layer by layer without needing of gradient computation after an initialization. Possible initialization schemes include utilizing the data matrix as initial weights and random initialization. The study is followed by an investigation into the representation capability and the output variance of the learning scheme. An extensive experimentation on synthetic and real-world data sets validates its numerical feasibility.