Information Losses in Neural Classifiers from Sampling
This addresses the issue of understanding and quantifying information loss in neural networks for researchers, providing incremental improvements over prior bounds.
The paper tackles the problem of information losses in neural classifiers due to finite training datasets, proving a relationship between these losses and the expected total variation of the model, and obtains bounds that are less sensitive to input compression and smaller than existing ones, with experimental validation.
This paper considers the subject of information losses arising from the finite datasets used in the training of neural classifiers. It proves a relationship between such losses as the product of the expected total variation of the estimated neural model with the information about the feature space contained in the hidden representation of that model. It then bounds this expected total variation as a function of the size of randomly sampled datasets in a fairly general setting, and without bringing in any additional dependence on model complexity. It ultimately obtains bounds on information losses that are less sensitive to input compression and in general much smaller than existing bounds. The paper then uses these bounds to explain some recent experimental findings of information compression in neural networks which cannot be explained by previous work. Finally, the paper shows that not only are these bounds much smaller than existing ones, but that they also correspond well with experiments.