A Neural Network Based on First Principles
This provides a theoretical foundation for neural network components, which is incremental as it builds on existing methods but offers new insights.
The authors derived a neural network from first principles using Maximum Entropy to find posterior distributions, resulting in activation functions like sigmoid and ReLU with theoretical justification, and proposed a theorem unifying special cases into an auto-encoder structure.
In this paper, a Neural network is derived from first principles, assuming only that each layer begins with a linear dimension-reducing transformation. The approach appeals to the principle of Maximum Entropy (MaxEnt) to find the posterior distribution of the input data of each layer, conditioned on the layer output variables. This posterior has a well-defined mean, the conditional mean estimator, that is calculated using a type of neural network with theoretically-derived activation functions similar to sigmoid, softplus, and relu. This implicitly provides a theoretical justification for their use. A theorem that finds the conditional distribution and conditional mean estimator under the MaxEnt prior is proposed, unifying results for special cases. Combining layers results in an auto-encoder with conventional feed-forward analysis network and a type of linear Bayesian belief network in the reconstruction path.