Singular leaning coefficients and efficiency in learning theory
This work addresses a foundational problem in machine learning theory for researchers, but it is incremental as it builds on early-stage analysis of singular models.
The paper tackled the theoretical analysis of learning efficiency in singular learning models, such as deep linear models and three-layer neural networks with ReLU units, by examining learning coefficients, and extended the results to include the Softmax function.
Singular learning models with non-positive Fisher information matrices include neural networks, reduced-rank regression, Boltzmann machines, normal mixture models, and others. These models have been widely used in the development of learning machines. However, theoretical analysis is still in its early stages. In this paper, we examine learning coefficients, which indicate the general learning efficiency of deep linear learning models and three-layer neural network models with ReLU units. Finally, we extend the results to include the case of the Softmax function.