Achieving Constraints in Neural Networks: A Stochastic Augmented Lagrangian Approach
This addresses the need for more flexible and efficient regularization in deep learning, particularly for white-box models requiring interpretability, though it appears incremental as it builds on existing constrained optimization methods.
The paper tackled the problem of regularizing deep neural networks by framing training as a constrained optimization problem, using a Stochastic Augmented Lagrangian method, and achieved higher accuracy and better constraint satisfaction on MNIST, CIFAR10, and CIFAR100 datasets.
Regularizing Deep Neural Networks (DNNs) is essential for improving generalizability and preventing overfitting. Fixed penalty methods, though common, lack adaptability and suffer from hyperparameter sensitivity. In this paper, we propose a novel approach to DNN regularization by framing the training process as a constrained optimization problem. Where the data fidelity term is the minimization objective and the regularization terms serve as constraints. Then, we employ the Stochastic Augmented Lagrangian (SAL) method to achieve a more flexible and efficient regularization mechanism. Our approach extends beyond black-box regularization, demonstrating significant improvements in white-box models, where weights are often subject to hard constraints to ensure interpretability. Experimental results on image-based classification on MNIST, CIFAR10, and CIFAR100 datasets validate the effectiveness of our approach. SAL consistently achieves higher Accuracy while also achieving better constraint satisfaction, thus showcasing its potential for optimizing DNNs under constrained settings.