Emergence of Invariance and Disentanglement in Deep Representations
This work addresses the fundamental challenge of improving generalization and interpretability in deep learning by linking invariance to information theory, though it is incremental as it builds on existing principles.
The paper tackles the problem of understanding invariance and disentanglement in deep representations by showing that invariance to nuisance factors is equivalent to information minimality, and proposes a regularization method based on bounding an overfitting term in the loss, which predicts and verifies sharp phase transitions in generalization error.
Using established principles from Statistics and Information Theory, we show that invariance to nuisance factors in a deep neural network is equivalent to information minimality of the learned representation, and that stacking layers and injecting noise during training naturally bias the network towards learning invariant representations. We then decompose the cross-entropy loss used during training and highlight the presence of an inherent overfitting term. We propose regularizing the loss by bounding such a term in two equivalent ways: One with a Kullbach-Leibler term, which relates to a PAC-Bayes perspective; the other using the information in the weights as a measure of complexity of a learned model, yielding a novel Information Bottleneck for the weights. Finally, we show that invariance and independence of the components of the representation learned by the network are bounded above and below by the information in the weights, and therefore are implicitly optimized during training. The theory enables us to quantify and predict sharp phase transitions between underfitting and overfitting of random labels when using our regularized loss, which we verify in experiments, and sheds light on the relation between the geometry of the loss function, invariance properties of the learned representation, and generalization error.