Information-Theoretic Local Minima Characterization and Regularization
This addresses the challenge of understanding and enhancing the performance of deep learning models for researchers and practitioners, offering a unified approach that is incremental but practical.
The paper tackles the problem of characterizing and improving generalizability across local minima in deep neural networks by proposing a metric based on Fisher information that serves both as an indicator of generalizability and as a practical regularizer, achieving strong empirical results on datasets like CIFAR-10, CIFAR-100, and ImageNet.
Recent advances in deep learning theory have evoked the study of generalizability across different local minima of deep neural networks (DNNs). While current work focused on either discovering properties of good local minima or developing regularization techniques to induce good local minima, no approach exists that can tackle both problems. We achieve these two goals successfully in a unified manner. Specifically, based on the observed Fisher information we propose a metric both strongly indicative of generalizability of local minima and effectively applied as a practical regularizer. We provide theoretical analysis including a generalization bound and empirically demonstrate the success of our approach in both capturing and improving the generalizability of DNNs. Experiments are performed on CIFAR-10, CIFAR-100 and ImageNet for various network architectures.