Flow of Information in Feed-Forward Deep Neural Networks
This work addresses the need for precise behavioral understanding in neural networks for efficient deployment, though it appears incremental as it builds on existing Information Bottleneck principles.
The paper tackled the problem of understanding information flow in feed-forward deep neural networks by using an information theoretic approach to analyze entropy changes between layers and applying the Information Bottleneck principle to develop a constrained optimization method for training, resulting in a derived lower bound for data representation with acceptable distortion.
Feed-forward deep neural networks have been used extensively in various machine learning applications. Developing a precise understanding of the underling behavior of neural networks is crucial for their efficient deployment. In this paper, we use an information theoretic approach to study the flow of information in a neural network and to determine how entropy of information changes between consecutive layers. Moreover, using the Information Bottleneck principle, we develop a constrained optimization problem that can be used in the training process of a deep neural network. Furthermore, we determine a lower bound for the level of data representation that can be achieved in a deep neural network with an acceptable level of distortion.