SHADE: Information-Based Regularization for Deep Learning
This work addresses regularization challenges for deep learning practitioners, offering a novel approach that could enhance model generalization, though it appears incremental as it builds on existing regularization concepts.
The authors tackled the problem of regularization in deep neural networks by proposing SHADE, an information-theory-based regularization scheme that uses conditional entropy as a prior to decouple invariant representation learning from data fitting, and they demonstrated improved classification performance compared to standard methods.
Regularization is a big issue for training deep neural networks. In this paper, we propose a new information-theory-based regularization scheme named SHADE for SHAnnon DEcay. The originality of the approach is to define a prior based on conditional entropy, which explicitly decouples the learning of invariant representations in the regularizer and the learning of correlations between inputs and labels in the data fitting term. Our second contribution is to derive a stochastic version of the regularizer compatible with deep learning, resulting in a tractable training scheme. We empirically validate the efficiency of our approach to improve classification performances compared to standard regularization schemes on several standard architectures.