Constrained Empirical Risk Minimization: Theory and Practice
This addresses the challenge of exact constraint enforcement in DNNs, which is incremental as it builds on existing methods but focuses on constraints outside geometric deep learning.
The paper tackles the problem of enforcing constraints on deep neural networks by proposing a framework that restricts parameters to a submanifold to satisfy constraints exactly during training, and demonstrates its application by using wavelet-constrained CNNs for medical contour prediction.
Deep Neural Networks (DNNs) are widely used for their ability to effectively approximate large classes of functions. This flexibility, however, makes the strict enforcement of constraints on DNNs an open problem. Here we present a framework that, under mild assumptions, allows the exact enforcement of constraints on parameterized sets of functions such as DNNs. Instead of imposing "soft'' constraints via additional terms in the loss, we restrict (a subset of) the DNN parameters to a submanifold on which the constraints are satisfied exactly throughout the entire training procedure. We focus on constraints that are outside the scope of equivariant networks used in Geometric Deep Learning. As a major example of the framework, we restrict filters of a Convolutional Neural Network (CNN) to be wavelets, and apply these wavelet networks to the task of contour prediction in the medical domain.