Bounded Rational Decision-Making in Feedforward Neural Networks
This work addresses overfitting in neural networks for practitioners, offering a novel regularization method, though it is incremental as it builds on existing information-theoretic frameworks.
The paper tackles the problem of overfitting in feedforward neural networks by modeling bounded rational decision-making as information-theoretic channels, deriving synaptic weight update rules that introduce regularization. In experiments on MNIST, this approach prevents overfitting across architectures and achieves competitive results with techniques like dropout and Bayes by backprop.
Bounded rational decision-makers transform sensory input into motor output under limited computational resources. Mathematically, such decision-makers can be modeled as information-theoretic channels with limited transmission rate. Here, we apply this formalism for the first time to multilayer feedforward neural networks. We derive synaptic weight update rules for two scenarios, where either each neuron is considered as a bounded rational decision-maker or the network as a whole. In the update rules, bounded rationality translates into information-theoretically motivated types of regularization in weight space. In experiments on the MNIST benchmark classification task for handwritten digits, we show that such information-theoretic regularization successfully prevents overfitting across different architectures and attains results that are competitive with other recent techniques like dropout, dropconnect and Bayes by backprop, for both ordinary and convolutional neural networks.