Understanding Autoencoders with Information Theoretic Concepts
This provides a theoretical framework for understanding autoencoders, which is incremental but addresses a foundational gap in deep learning theory.
The paper tackles the lack of theoretical understanding of deep neural networks by applying information theory to analyze autoencoders, revealing properties like layer-wise information flow and bottleneck dimensionality that impact optimal design and generalization.
Despite their great success in practical applications, there is still a lack of theoretical and systematic methods to analyze deep neural networks. In this paper, we illustrate an advanced information theoretic methodology to understand the dynamics of learning and the design of autoencoders, a special type of deep learning architectures that resembles a communication channel. By generalizing the information plane to any cost function, and inspecting the roles and dynamics of different layers using layer-wise information quantities, we emphasize the role that mutual information plays in quantifying learning from data. We further suggest and also experimentally validate, for mean square error training, three fundamental properties regarding the layer-wise flow of information and intrinsic dimensionality of the bottleneck layer, using respectively the data processing inequality and the identification of a bifurcation point in the information plane that is controlled by the given data. Our observations have a direct impact on the optimal design of autoencoders, the design of alternative feedforward training methods, and even in the problem of generalization.