Exploiting Explainable Metrics for Augmented SGD
This work addresses the challenge of understanding and improving generalization in deep learning for researchers and practitioners, though it is incremental as it builds on existing SGD methods.
The authors tackled the problem of quantifying learning quality in deep neural network layers by proposing new explainability metrics based on low-rank factorization, which they used to adaptively adjust learning rates per layer in SGD, resulting in improved generalization performance across applications, architectures, and datasets.
Explaining the generalization characteristics of deep learning is an emerging topic in advanced machine learning. There are several unanswered questions about how learning under stochastic optimization really works and why certain strategies are better than others. In this paper, we address the following question: \textit{can we probe intermediate layers of a deep neural network to identify and quantify the learning quality of each layer?} With this question in mind, we propose new explainability metrics that measure the redundant information in a network's layers using a low-rank factorization framework and quantify a complexity measure that is highly correlated with the generalization performance of a given optimizer, network, and dataset. We subsequently exploit these metrics to augment the Stochastic Gradient Descent (SGD) optimizer by adaptively adjusting the learning rate in each layer to improve in generalization performance. Our augmented SGD -- dubbed RMSGD -- introduces minimal computational overhead compared to SOTA methods and outperforms them by exhibiting strong generalization characteristics across application, architecture, and dataset.