The training accuracy of two-layer neural networks: its estimation and understanding using random datasets
This study addresses the fundamental challenge of transparency in deep learning for researchers, though it is incremental as it focuses on theoretical estimation without real-world generalization analysis.
The authors tackled the problem of understanding neural network mechanisms by proposing a theory to estimate training accuracy for two-layer neural networks on random datasets without training, using only dimensionality, number of inputs, and neurons, and verified it experimentally with results indicating applicability across dimensions and potential extension to deeper models.
Although the neural network (NN) technique plays an important role in machine learning, understanding the mechanism of NN models and the transparency of deep learning still require more basic research. In this study, we propose a novel theory based on space partitioning to estimate the approximate training accuracy for two-layer neural networks on random datasets without training. There appear to be no other studies that have proposed a method to estimate training accuracy without using input data and/or trained models. Our method estimates the training accuracy for two-layer fully-connected neural networks on two-class random datasets using only three arguments: the dimensionality of inputs (d), the number of inputs (N), and the number of neurons in the hidden layer (L). We have verified our method using real training accuracies in our experiments. The results indicate that the method will work for any dimension, and the proposed theory could extend also to estimate deeper NN models. The main purpose of this paper is to understand the mechanism of NN models by the approach of estimating training accuracy but not to analyze their generalization nor their performance in real-world applications. This study may provide a starting point for a new way for researchers to make progress on the difficult problem of understanding deep learning.