Correlation between entropy and generalizability in a neural network
This work addresses the theoretical gap in explaining neural network generalizability for researchers, but it is incremental as it applies to a limited, simple case.
The study tackled the problem of understanding why neural networks generalize by calculating entropy in parameter space using the Wang-Landau Monte Carlo algorithm, finding that entropical forces aid generalizability, though results were based on a simple spiral dataset and small network.
Although neural networks can solve very complex machine-learning problems, the theoretical reason for their generalizability is still not fully understood. Here we use Wang-Landau Mote Carlo algorithm to calculate the entropy (logarithm of the volume of a part of the parameter space) at a given test accuracy, and a given training loss function value or training accuracy. Our results show that entropical forces help generalizability. Although our study is on a very simple application of neural networks (a spiral dataset and a small, fully-connected neural network), our approach should be useful in explaining the generalizability of more complicated neural networks in future works.